README.md

---
license: apache-2.0
library_name: transformers
base_model: Qwen/Qwen3.8-27B
base_model_relation: finetune
pipeline_tag: image-text-to-text
tags:
- agentic
- smaug
- abacusai
---
<div align="center">
  <picture>
      <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/63128dd099791aa61d180c72/dJ6uR23m09M-YEafBpmea.png" width="20%" alt="Smaug-Mini">
  </picture>
</div>
<hr>
<div align="center" style="line-height:1">
  <a href="https://abacus.ai" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Abacus.AI-white?color=1783ff&logoColor=white"/></a>
  <a href="https://huggingface.co/abacusai" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Abacus.AI-ffc107?color=ffc107&logoColor=white"/></a>
</div>
<div align="center" style="line-height: 1;">
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-f5de53?&color=f5de53"/></a>
</div>

## 1. Model Introduction

Smaug-Mini is an agentic finetune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), finetuned by Abacus.AI. It improves end-to-end agentic tool use and automation performance (+4.5 on AutomationBench, +17.1 on JobBench, +2.0 overall on LiveBench, +2.5 on IFBench) while holding the base model's general capabilities at parity (GPQA-diamond at the official number, LiveBench coding and agentic-coding at board parity). The weights load exactly like the official release: same layout, same `Qwen3_5ForConditionalGeneration` architecture with the vision tower intact, same 262,144-token context, same reasoning interface with `xhigh` / `medium` / `low` effort levels.

This card describes the training approach and the evaluation results. Dataset contents are not disclosed; training data consists of multi-turn, tool-using automation episodes generated and verified against executable environments.

## 2. Model Summary

<div align="center">
<table>
<tbody>
<tr><td align="center"><strong>Architecture</strong></td><td align="center">Dense hybrid-attention transformer + vision tower</td></tr>
<tr><td align="center"><strong>Total Parameters</strong></td><td align="center">27B</td></tr>
<tr><td align="center"><strong>Number of Layers</strong></td><td align="center">64 (48 linear-attention + 16 full-attention, 3:1 interleave)</td></tr>
<tr><td align="center"><strong>Attention Mechanism</strong></td><td align="center">Gated linear attention &amp; full attention (GQA)</td></tr>
<tr><td align="center"><strong>Hidden Dimension</strong></td><td align="center">5120</td></tr>
<tr><td align="center"><strong>Number of Attention Heads</strong></td><td align="center">24 (4 KV heads)</td></tr>
<tr><td align="center"><strong>Vision Encoder</strong></td><td align="center">27-layer ViT, patch 16</td></tr>
<tr><td align="center"><strong>Vocabulary Size</strong></td><td align="center">~248K</td></tr>
<tr><td align="center"><strong>Context Length</strong></td><td align="center">262,144</td></tr>
<tr><td align="center"><strong>Multi-Token Prediction</strong></td><td align="center">1-layer MTP head (inherited; leave speculative decoding off)</td></tr>
<tr><td align="center"><strong>Precision</strong></td><td align="center">bfloat16</td></tr>
<tr><td align="center"><strong>Modality</strong></td><td align="center">Text, Image</td></tr>
<tr><td align="center"><strong>Base Model</strong></td><td align="center"><code>Qwen/Qwen3.8-27B</code></td></tr>
<tr><td align="center"><strong>Adaptation</strong></td><td align="center">On-policy RL (GRPO), LoRA merged as full delta (language trunk only)</td></tr>
</tbody>
</table>
</div>

## 3. Evaluation

### Smaug-Mini vs its base

*Higher is better. Base points from the Qwen3.8-27B model card and livebench.ai.*

<div style="max-width:1100px;margin:0 auto">
<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
<thead><tr>
<th style="padding:10px 6px;text-align:left;border-bottom:2px solid #2a78d6;width:26%"></th>
<th style="padding:10px 4px;text-align:center;font-weight:600;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;background:rgba(42,120,214,0.10);">Smaug-Mini</th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.8-27B<div style="font-size:10px;font-weight:400;color:#6B6B6B">(base)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.6-27B</th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.7-Plus</th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Opus4.6 Max</th>
</tr></thead>
<tbody>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Agentic</td></tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">AutomationBench</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>41.8</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.38)">37.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">&mdash;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">20.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">25.5</td>
</tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">JobBench</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>50.5</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">33.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">21.8</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.13)">27.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.27)">36.9</td>
</tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench agentic coding</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.46)">60.8</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>61.4</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">39.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">&mdash;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">49.0</td>
</tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">NL2Repo-Bench</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>55.8</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.18)">42.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">36.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">41.1</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.29)">47.6</td>
</tr>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Reasoning, knowledge &amp; instruction following</td></tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">GPQA-diamond</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.24)">89.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">89.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">87.8</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.35)">90.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>91.3</strong></td>
</tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">HLE</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.32)">34.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">30.8</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">24.0</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.33)">34.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>40.0</strong></td>
</tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">IFBench</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>82.0</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">79.5</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.19)">69.1</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.41)">79.1</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">62.5</td>
</tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench overall</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>76.9</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">75.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">64.0</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">&mdash;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.39)">74.5</td>
</tr>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Vision</td></tr>
<tr>
<td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">MMMU-Pro</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.10)">75.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.16)">76.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>80</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td>
</tr>
</tbody></table>
<div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card unless noted; the Opus 4.6 AutomationBench score is from our own harness run and its JobBench score is a reported number, not our run; LiveBench rows use the public <a href="https://livebench.ai">livebench.ai</a> leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from <a href="https://artificialanalysis.ai">Artificial Analysis</a> (its Opus 4.6 MMMU-Pro entry runs at max effort); &mdash; = not reported.</div>
</div>

### LiveBench category profile

*Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published [livebench.ai](https://livebench.ai) leaderboard entries (Smaug-Mini listed under the finetunes filter).*

<div align="center">
  <img src="figures/smaug_mini_livebench_categories.png" width="100%" alt="Smaug-Mini LiveBench category profile vs Qwen3.8-27B board entry">
</div>

<details>
<summary><b>Notes</b></summary>

All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort `xhigh`, with generation budgets large enough that no score is truncation-bound.

Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge.

</details>

## 4. Training Approach

Smaug-Mini was trained to make multi-turn tool use and automation episodes more reliable end to end. It is trained with on-policy reinforcement learning (GRPO) over multi-turn, tool-using automation episodes with verified, outcome-based rewards - an episode is rewarded for measurably completing its task in the environment. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta; the vision tower and multimodal projection are bitwise-identical to the base release.

One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.

## 5. Known Behaviors and Limitations

Smaug-Mini redistributes deliberation rather than adding it: more reasoning per action but fewer actions per episode, finishing passing episodes about three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%. Refusals are zero for both models, and the general-capability benchmarks in §3 show the deliberation shift does not come at the cost of base skills.

## 6. Deployment

Smaug-Mini serves as a drop-in replacement for Qwen3.8-27B on any stack that supports the base model. Reference vLLM invocation:

```bash
vllm serve abacusai/Smaug-Mini \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

Recommended sampling for agentic use: temperature 1.0, top_p 0.95, reasoning effort `xhigh` (the default). Recent vLLM versions return the chain-of-thought in the `reasoning` field of the message; when replaying conversation history, pass reasoning back under both `reasoning` and `reasoning_content` keys for portability across serving versions. Leave MTP-based speculative decoding off (§4).

## 7. License

Apache 2.0, inherited from Qwen3.8-27B.

## 8. Citation

```bibtex
@misc{abacusai2026smaugmini,
  title  = {Smaug-Mini},
  author = {Abacus.AI},
  year   = {2026},
  note   = {Agentic fine-tune of Qwen/Qwen3.8-27B},
  url    = {https://huggingface.co/abacusai/Smaug-Mini}
}
```

The Smaug line and the DPO-Positive method behind it:

```bibtex
@article{pal2024smaug,
  title={Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive},
  author={Pal, Arka and Karkhanis, Deep and Dooley, Samuel and
          Roberts, Manley and Naidu, Siddartha and White, Colin},
  journal={arXiv preprint arXiv:2402.13228},
  year={2024}
}
```

## 9. Contact Us

If you have any questions, please reach out at [Abacus.AI](https://abacus.ai).