--- library_name: transformers pipeline_tag: text-generation base_model: muradil211/AetherSearch_SFT datasets: - muradil211/AetherSearch_DPO tags: - aethersearch - agentic-search - search-augmented-generation - direct-preference-optimization - dpo - conversational - qwen2 language: - en ---
AetherSearch monogram # ๐Ÿ”ญ AetherSearch DPO ### A preference-aligned search agent for retrieval-augmented reasoning Built on **AetherSearch SFT** and aligned with Direct Preference Optimization (DPO) over **2,126 search-trajectory preference pairs**.

Training method: DPO Training pairs: 2,126 Context window: 32K

[๐Ÿ  Project](https://github.com/Muradil-mamat-211/AetherSearch) ยท [๐Ÿง  SFT checkpoint](https://huggingface.co/muradil211/AetherSearch_SFT)
> ๐Ÿ”Œ **Bring your own retriever.** AetherSearch DPO is a search-agent policy, > not a self-contained question-answering service. The host runtime must > execute each `...` request and return evidence inside > `...`. ## โœจ Highlights - ๐ŸŽฏ **Preference aligned** โ€” favors stronger search decisions, evidence use, and final answers over paired alternatives. - ๐Ÿ”Ž **Search native** โ€” produces explicit reasoning, retrieval requests, and grounded answers using the AetherSearch XML protocol. - ๐Ÿ” **Multi-step retrieval** โ€” can request additional evidence when an existing trajectory is insufficient. - ๐Ÿงพ **Retrieval-aware DPO** โ€” retrieved information remains visible as context but is excluded from preference-loss scoring. ## ๐Ÿง  How it works ```text Question โ”‚ โ–ผ reason about what is missing โ”‚ โ–ผ focused retrieval query โ”€โ”€โ”€โ”€โ”€โ–บ Search / RAG backend โ–ฒ โ”‚ โ””โ”€โ”€โ”€โ”€ retrieved evidence โ—„โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”œโ”€โ”€ repeat the search loop when more evidence is needed โ–ผ evidence-grounded final answer ``` The model produces reasoning, search, and answer spans. The surrounding runtime parses each completed `` span, runs retrieval, appends the result as ``, and resumes generation until the model emits an `` span. ## ๐Ÿ“Š Model at a glance | Field | Value | |---|---| | ๐Ÿงฌ Base checkpoint | [`muradil211/AetherSearch_SFT`](https://huggingface.co/muradil211/AetherSearch_SFT) | | ๐Ÿงฌ Base revision | `437aca474d3966e57e82af565db95d0ad64aa24d` | | ๐Ÿ—๏ธ Architecture | Qwen2 causal language model | | ๐Ÿ”ข Parameters | 3,085,938,688 | | ๐ŸŽ›๏ธ Weight dtype | BF16 | | ๐Ÿ“ Context window | 32,768 positions; training sequences capped at 4,096 | | ๐ŸŽฏ Alignment method | Direct Preference Optimization | ## ๐ŸŽฏ DPO alignment Supervised fine-tuning teaches the model how to follow the search protocol; DPO then teaches it which of two valid-looking continuations is preferable. For a shared prompt \\(x\\), preferred continuation \\(y_w\\), rejected continuation \\(y_l\\), policy \\(\pi_\theta\\), and frozen SFT reference \\(\pi_{\mathrm{ref}}\\), training minimizes: $$ \mathcal{L}_{\mathrm{DPO}} = -\log \sigma\!\left( \beta \left[ \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)} \right] \right). $$ In plain terms, the policy learns to widen the preference margin between the chosen and rejected search trajectories while the frozen SFT model anchors the update. This directly optimizes pairwise preferences without training a separate reward model or running an online RL loop. The loss is adapted to the agent-environment boundary: - Shared prompt tokens are masked on both sides. - Environment-provided `...` spans inside either continuation are masked while remaining visible as context. - All remaining assistant tokens contribute to summed sequence log probabilities. - Answer-terminal continuations include the final `<|im_end|>` token; search-terminal continuations stop before it so the runtime can insert the next retrieval result. Both the initial policy and frozen reference use the pinned AetherSearch SFT checkpoint. The released run uses \\(\beta=0.1\\). ## ๐Ÿ› ๏ธ Training recipe | Setting | Value | Setting | Value | |---|---:|---|---:| | Epochs | 1 | Learning rate | `5e-7` | | DPO beta | `0.1` | Scheduler | Cosine | | Global batch size | 12 pairs | Per-device batch | 1 pair | | Precision | BF16 | Max sequence length | 4,096 | | Warmup ratio | `0.03` | Weight decay | `0.0` | | Distributed optimizer | DeepSpeed ZeRO-3 | Seed | 42 | The policy and frozen reference are both sharded with ZeRO-3; gradients and optimizer updates are applied only to the policy. ## ๐Ÿš€ Quick start ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "muradil211/AetherSearch_DPO" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) model.config.use_cache = True ``` Loading the checkpoint is only the first step. For end-to-end use, stop generation after each complete `` request, execute it with your retriever, append the result as ``, and resume generation. Stop when the model emits a complete `` span, and preserve the XML protocol exactly throughout the loop. ## โš ๏ธ Limitations - DPO improves fit to the supplied preference pairs; it does not guarantee factual, supported, or safe outputs. - Preference labels include curated hard negatives and trajectory corrections, not human preference votes for every pair. - Retrieval quality, evidence validation, answer verification, and deployment safeguards remain the caller's responsibility. ## ๐Ÿ“œ Terms No additional blanket license is asserted here. Review the [Qwen2.5-3B-Instruct license](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE), the [AetherSearch SFT terms](https://huggingface.co/muradil211/AetherSearch_SFT), and the [AetherSearch DPO data attribution](https://huggingface.co/datasets/muradil211/AetherSearch_DPO/blob/main/ATTRIBUTION.md) before redistribution or downstream use. ---
**Built for agentic search and retrieval-augmented reasoning.** ๐Ÿ”Žโœจ