Instructions to use muradil211/AetherSearch_DPO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use muradil211/AetherSearch_DPO with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="muradil211/AetherSearch_DPO")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("muradil211/AetherSearch_DPO")
model = AutoModelForCausalLM.from_pretrained("muradil211/AetherSearch_DPO", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use muradil211/AetherSearch_DPO with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "muradil211/AetherSearch_DPO"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "muradil211/AetherSearch_DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/muradil211/AetherSearch_DPO

SGLang

How to use muradil211/AetherSearch_DPO with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "muradil211/AetherSearch_DPO" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "muradil211/AetherSearch_DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "muradil211/AetherSearch_DPO" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "muradil211/AetherSearch_DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use muradil211/AetherSearch_DPO with Docker Model Runner:
```
docker model run hf.co/muradil211/AetherSearch_DPO
```

AetherSearch_DPO / README.md

muradil211

Use Hub-compatible inline KaTeX delimiters

1ca93d0 verified 2 months ago

preview code

Raw History Blame Contribute Delete

6.75 kB

---
library_name: transformers
pipeline_tag: text-generation
base_model: muradil211/AetherSearch_SFT
datasets:
  - muradil211/AetherSearch_DPO
tags:
  - aethersearch
  - agentic-search
  - search-augmented-generation
  - direct-preference-optimization
  - dpo
  - conversational
  - qwen2
language:
  - en
---
<div align="center">

<img src="assets/aethersearch-mark.svg" alt="AetherSearch monogram" width="144">

# 🔭 AetherSearch DPO

### A preference-aligned search agent for retrieval-augmented reasoning

Built on **AetherSearch SFT** and aligned with Direct Preference Optimization
(DPO) over **2,126 search-trajectory preference pairs**.

<p>
  <img src="https://img.shields.io/badge/Method-DPO-0F766E?style=flat-square" alt="Training method: DPO">
  <img src="https://img.shields.io/badge/Pairs-2%2C126-F59E0B?style=flat-square" alt="Training pairs: 2,126">
  <img src="https://img.shields.io/badge/Context-32K-2563EB?style=flat-square" alt="Context window: 32K">
</p>

[🏠 Project](https://github.com/Muradil-mamat-211/AetherSearch) ·
[🧠 SFT checkpoint](https://huggingface.co/muradil211/AetherSearch_SFT)

</div>

> 🔌 **Bring your own retriever.** AetherSearch DPO is a search-agent policy,
> not a self-contained question-answering service. The host runtime must
> execute each `<search>...</search>` request and return evidence inside
> `<information>...</information>`.

## ✨ Highlights

- 🎯 **Preference aligned** — favors stronger search decisions, evidence use,
  and final answers over paired alternatives.
- 🔎 **Search native** — produces explicit reasoning, retrieval requests, and
  grounded answers using the AetherSearch XML protocol.
- 🔁 **Multi-step retrieval** — can request additional evidence when an
  existing trajectory is insufficient.
- 🧾 **Retrieval-aware DPO** — retrieved information remains visible as context
  but is excluded from preference-loss scoring.

## 🧠 How it works

```text
Question
   │
   ▼
<think>reason about what is missing</think>
   │
   ▼
<search>focused retrieval query</search> ─────► Search / RAG backend
   ▲                                                  │
   └──── <information>retrieved evidence</information> ◄────┘
   │
   ├── repeat the search loop when more evidence is needed
   ▼
<answer>evidence-grounded final answer</answer>
```

The model produces reasoning, search, and answer spans. The surrounding
runtime parses each completed `<search>` span, runs retrieval, appends the
result as `<information>`, and resumes generation until the model emits an
`<answer>` span.

## 📊 Model at a glance

| Field | Value |
|---|---|
| 🧬 Base checkpoint | [`muradil211/AetherSearch_SFT`](https://huggingface.co/muradil211/AetherSearch_SFT) |
| 🧬 Base revision | `437aca474d3966e57e82af565db95d0ad64aa24d` |
| 🏗️ Architecture | Qwen2 causal language model |
| 🔢 Parameters | 3,085,938,688 |
| 🎛️ Weight dtype | BF16 |
| 📏 Context window | 32,768 positions; training sequences capped at 4,096 |
| 🎯 Alignment method | Direct Preference Optimization |

## 🎯 DPO alignment

Supervised fine-tuning teaches the model how to follow the search protocol;
DPO then teaches it which of two valid-looking continuations is preferable.
For a shared prompt \\(x\\), preferred continuation \\(y_w\\), rejected
continuation \\(y_l\\), policy \\(\pi_\theta\\), and frozen SFT reference
\\(\pi_{\mathrm{ref}}\\), training minimizes:

$$
\mathcal{L}_{\mathrm{DPO}} = -\log \sigma\!\left(
\beta \left[
\log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)}
-
\log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}
\right]
\right).
$$
In plain terms, the policy learns to widen the preference margin between the
chosen and rejected search trajectories while the frozen SFT model anchors the
update. This directly optimizes pairwise preferences without training a
separate reward model or running an online RL loop.
The loss is adapted to the agent-environment boundary:
- Shared prompt tokens are masked on both sides.
- Environment-provided `<information>...</information>` spans inside either
  continuation are masked while remaining visible as context.
- All remaining assistant tokens contribute to summed sequence log
  probabilities.
- Answer-terminal continuations include the final `<|im_end|>` token;
  search-terminal continuations stop before it so the runtime can insert the
  next retrieval result.

Both the initial policy and frozen reference use the pinned AetherSearch SFT
checkpoint. The released run uses \\(\beta=0.1\\).

## 🛠️ Training recipe

| Setting | Value | Setting | Value |
|---|---:|---|---:|
| Epochs | 1 | Learning rate | `5e-7` |
| DPO beta | `0.1` | Scheduler | Cosine |
| Global batch size | 12 pairs | Per-device batch | 1 pair |
| Precision | BF16 | Max sequence length | 4,096 |
| Warmup ratio | `0.03` | Weight decay | `0.0` |
| Distributed optimizer | DeepSpeed ZeRO-3 | Seed | 42 |

The policy and frozen reference are both sharded with ZeRO-3; gradients and
optimizer updates are applied only to the policy.

## 🚀 Quick start

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "muradil211/AetherSearch_DPO"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.config.use_cache = True
```

Loading the checkpoint is only the first step. For end-to-end use, stop
generation after each complete `<search>` request, execute it with your
retriever, append the result as `<information>`, and resume generation. Stop
when the model emits a complete `<answer>` span, and preserve the XML protocol
exactly throughout the loop.

## ⚠️ Limitations

- DPO improves fit to the supplied preference pairs; it does not guarantee
  factual, supported, or safe outputs.
- Preference labels include curated hard negatives and trajectory corrections,
  not human preference votes for every pair.
- Retrieval quality, evidence validation, answer verification, and deployment
  safeguards remain the caller's responsibility.

## 📜 Terms

No additional blanket license is asserted here. Review the
[Qwen2.5-3B-Instruct license](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE),
the [AetherSearch SFT terms](https://huggingface.co/muradil211/AetherSearch_SFT),
and the [AetherSearch DPO data attribution](https://huggingface.co/datasets/muradil211/AetherSearch_DPO/blob/main/ATTRIBUTION.md)
before redistribution or downstream use.

---

<div align="center">

**Built for agentic search and retrieval-augmented reasoning.** 🔎✨

</div>

	---
	library_name: transformers
	pipeline_tag: text-generation
	base_model: muradil211/AetherSearch_SFT
	datasets:
	- muradil211/AetherSearch_DPO
	tags:
	- aethersearch
	- agentic-search
	- search-augmented-generation
	- direct-preference-optimization
	- dpo
	- conversational
	- qwen2
	language:
	- en
	---

	<div align="center">

	<img src="assets/aethersearch-mark.svg" alt="AetherSearch monogram" width="144">

	# 🔭 AetherSearch DPO

	### A preference-aligned search agent for retrieval-augmented reasoning

	Built on AetherSearch SFT and aligned with Direct Preference Optimization
	(DPO) over 2,126 search-trajectory preference pairs.

	<p>
	<img src="https://img.shields.io/badge/Method-DPO-0F766E?style=flat-square" alt="Training method: DPO">
	<img src="https://img.shields.io/badge/Pairs-2%2C126-F59E0B?style=flat-square" alt="Training pairs: 2,126">
	<img src="https://img.shields.io/badge/Context-32K-2563EB?style=flat-square" alt="Context window: 32K">
	</p>

	[🏠 Project](https://github.com/Muradil-mamat-211/AetherSearch) ·
	[🧠 SFT checkpoint](https://huggingface.co/muradil211/AetherSearch_SFT)

	</div>

	> 🔌 Bring your own retriever. AetherSearch DPO is a search-agent policy,
	> not a self-contained question-answering service. The host runtime must
	> execute each `<search>...</search>` request and return evidence inside
	> `<information>...</information>`.

	## ✨ Highlights

	- 🎯 Preference aligned — favors stronger search decisions, evidence use,
	and final answers over paired alternatives.
	- 🔎 Search native — produces explicit reasoning, retrieval requests, and
	grounded answers using the AetherSearch XML protocol.
	- 🔁 Multi-step retrieval — can request additional evidence when an
	existing trajectory is insufficient.
	- 🧾 Retrieval-aware DPO — retrieved information remains visible as context
	but is excluded from preference-loss scoring.

	## 🧠 How it works

	```text
	Question
	│
	▼
	<think>reason about what is missing</think>
	│
	▼
	<search>focused retrieval query</search> ─────► Search / RAG backend
	▲ │
	└──── <information>retrieved evidence</information> ◄────┘
	│
	├── repeat the search loop when more evidence is needed
	▼
	<answer>evidence-grounded final answer</answer>
	```

	The model produces reasoning, search, and answer spans. The surrounding
	runtime parses each completed `<search>` span, runs retrieval, appends the
	result as `<information>`, and resumes generation until the model emits an
	`<answer>` span.

	## 📊 Model at a glance

	\| Field \| Value \|
	\|---\|---\|
	\| 🧬 Base checkpoint \| [`muradil211/AetherSearch_SFT`](https://huggingface.co/muradil211/AetherSearch_SFT) \|
	\| 🧬 Base revision \| `437aca474d3966e57e82af565db95d0ad64aa24d` \|
	\| 🏗️ Architecture \| Qwen2 causal language model \|
	\| 🔢 Parameters \| 3,085,938,688 \|
	\| 🎛️ Weight dtype \| BF16 \|
	\| 📏 Context window \| 32,768 positions; training sequences capped at 4,096 \|
	\| 🎯 Alignment method \| Direct Preference Optimization \|

	## 🎯 DPO alignment

	Supervised fine-tuning teaches the model how to follow the search protocol;
	DPO then teaches it which of two valid-looking continuations is preferable.
	For a shared prompt \\(x\\), preferred continuation \\(y_w\\), rejected
	continuation \\(y_l\\), policy \\(\pi_\theta\\), and frozen SFT reference
	\\(\pi_{\mathrm{ref}}\\), training minimizes:

	$$
	\mathcal{L}_{\mathrm{DPO}} = -\log \sigma\!\left(
	\beta \left[
	\log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)}
	-
	\log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}
	\right]
	\right).
	$$

	In plain terms, the policy learns to widen the preference margin between the
	chosen and rejected search trajectories while the frozen SFT model anchors the
	update. This directly optimizes pairwise preferences without training a
	separate reward model or running an online RL loop.

	The loss is adapted to the agent-environment boundary:

	- Shared prompt tokens are masked on both sides.
	- Environment-provided `<information>...</information>` spans inside either
	continuation are masked while remaining visible as context.
	- All remaining assistant tokens contribute to summed sequence log
	probabilities.
	- Answer-terminal continuations include the final `<\|im_end\|>` token;
	search-terminal continuations stop before it so the runtime can insert the
	next retrieval result.

	Both the initial policy and frozen reference use the pinned AetherSearch SFT
	checkpoint. The released run uses \\(\beta=0.1\\).

	## 🛠️ Training recipe

	\| Setting \| Value \| Setting \| Value \|
	\|---\|---:\|---\|---:\|
	\| Epochs \| 1 \| Learning rate \| `5e-7` \|
	\| DPO beta \| `0.1` \| Scheduler \| Cosine \|
	\| Global batch size \| 12 pairs \| Per-device batch \| 1 pair \|
	\| Precision \| BF16 \| Max sequence length \| 4,096 \|
	\| Warmup ratio \| `0.03` \| Weight decay \| `0.0` \|
	\| Distributed optimizer \| DeepSpeed ZeRO-3 \| Seed \| 42 \|

	The policy and frozen reference are both sharded with ZeRO-3; gradients and
	optimizer updates are applied only to the policy.

	## 🚀 Quick start

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model_id = "muradil211/AetherSearch_DPO"

	tokenizer = AutoTokenizer.from_pretrained(model_id)
	model = AutoModelForCausalLM.from_pretrained(
	model_id,
	torch_dtype=torch.bfloat16,
	device_map="auto",
	)
	model.config.use_cache = True
	```

	Loading the checkpoint is only the first step. For end-to-end use, stop
	generation after each complete `<search>` request, execute it with your
	retriever, append the result as `<information>`, and resume generation. Stop
	when the model emits a complete `<answer>` span, and preserve the XML protocol
	exactly throughout the loop.

	## ⚠️ Limitations

	- DPO improves fit to the supplied preference pairs; it does not guarantee
	factual, supported, or safe outputs.
	- Preference labels include curated hard negatives and trajectory corrections,
	not human preference votes for every pair.
	- Retrieval quality, evidence validation, answer verification, and deployment
	safeguards remain the caller's responsibility.

	## 📜 Terms

	No additional blanket license is asserted here. Review the
	[Qwen2.5-3B-Instruct license](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE),
	the [AetherSearch SFT terms](https://huggingface.co/muradil211/AetherSearch_SFT),
	and the [AetherSearch DPO data attribution](https://huggingface.co/datasets/muradil211/AetherSearch_DPO/blob/main/ATTRIBUTION.md)
	before redistribution or downstream use.

	---

	<div align="center">

	Built for agentic search and retrieval-augmented reasoning. 🔎✨

	</div>