| --- |
| license: apache-2.0 |
| library_name: transformers |
| base_model: Qwen/Qwen3.8-27B |
| base_model_relation: finetune |
| pipeline_tag: image-text-to-text |
| tags: |
| - agentic |
| - smaug |
| - abacusai |
| --- |
| <div align="center"> |
| <picture> |
| <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/63128dd099791aa61d180c72/dJ6uR23m09M-YEafBpmea.png" width="20%" alt="Smaug-Mini"> |
| </picture> |
| </div> |
| <hr> |
| <div align="center" style="line-height:1"> |
| <a href="https://abacus.ai" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Abacus.AI-white?color=1783ff&logoColor=white"/></a> |
| <a href="https://huggingface.co/abacusai" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Abacus.AI-ffc107?color=ffc107&logoColor=white"/></a> |
| </div> |
| <div align="center" style="line-height: 1;"> |
| <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-f5de53?&color=f5de53"/></a> |
| </div> |
| |
| ## 1. Model Introduction |
|
|
| Smaug-Mini is an agentic finetune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), finetuned by Abacus.AI. It improves end-to-end agentic tool use and automation performance (+4.5 on AutomationBench, +17.1 on JobBench, +2.0 overall on LiveBench, +2.5 on IFBench) while holding the base model's general capabilities at parity (GPQA-diamond at the official number, LiveBench coding and agentic-coding at board parity). The weights load exactly like the official release: same layout, same `Qwen3_5ForConditionalGeneration` architecture with the vision tower intact, same 262,144-token context, same reasoning interface with `xhigh` / `medium` / `low` effort levels. |
|
|
| This card describes the training approach and the evaluation results. Dataset contents are not disclosed; training data consists of multi-turn, tool-using automation episodes generated and verified against executable environments. |
|
|
| ## 2. Model Summary |
|
|
| <div align="center"> |
| <table> |
| <tbody> |
| <tr><td align="center"><strong>Architecture</strong></td><td align="center">Dense hybrid-attention transformer + vision tower</td></tr> |
| <tr><td align="center"><strong>Total Parameters</strong></td><td align="center">27B</td></tr> |
| <tr><td align="center"><strong>Number of Layers</strong></td><td align="center">64 (48 linear-attention + 16 full-attention, 3:1 interleave)</td></tr> |
| <tr><td align="center"><strong>Attention Mechanism</strong></td><td align="center">Gated linear attention & full attention (GQA)</td></tr> |
| <tr><td align="center"><strong>Hidden Dimension</strong></td><td align="center">5120</td></tr> |
| <tr><td align="center"><strong>Number of Attention Heads</strong></td><td align="center">24 (4 KV heads)</td></tr> |
| <tr><td align="center"><strong>Vision Encoder</strong></td><td align="center">27-layer ViT, patch 16</td></tr> |
| <tr><td align="center"><strong>Vocabulary Size</strong></td><td align="center">~248K</td></tr> |
| <tr><td align="center"><strong>Context Length</strong></td><td align="center">262,144</td></tr> |
| <tr><td align="center"><strong>Multi-Token Prediction</strong></td><td align="center">1-layer MTP head (inherited; leave speculative decoding off)</td></tr> |
| <tr><td align="center"><strong>Precision</strong></td><td align="center">bfloat16</td></tr> |
| <tr><td align="center"><strong>Modality</strong></td><td align="center">Text, Image</td></tr> |
| <tr><td align="center"><strong>Base Model</strong></td><td align="center"><code>Qwen/Qwen3.8-27B</code></td></tr> |
| <tr><td align="center"><strong>Adaptation</strong></td><td align="center">On-policy RL (GRPO), LoRA merged as full delta (language trunk only)</td></tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| ## 3. Evaluation |
|
|
| ### Smaug-Mini vs its base |
|
|
| *Higher is better. Base points from the Qwen3.8-27B model card and livebench.ai.* |
|
|
| <div style="max-width:1100px;margin:0 auto"> |
| <table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px"> |
| <thead><tr> |
| <th style="padding:10px 6px;text-align:left;border-bottom:2px solid #2a78d6;width:26%"></th> |
| <th style="padding:10px 4px;text-align:center;font-weight:600;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;background:rgba(42,120,214,0.10);">Smaug-Mini</th> |
| <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.8-27B<div style="font-size:10px;font-weight:400;color:#6B6B6B">(base)</div></th> |
| <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.6-27B</th> |
| <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Qwen3.7-Plus</th> |
| <th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Opus4.6 Max</th> |
| </tr></thead> |
| <tbody> |
| <tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Agentic</td></tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">AutomationBench</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>41.8</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.38)">37.3</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">20.4</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">25.5</td> |
| </tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">JobBench</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>50.5</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">33.4</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">21.8</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.13)">27.6</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.27)">36.9</td> |
| </tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench agentic coding</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.46)">60.8</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>61.4</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">39.3</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">49.0</td> |
| </tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">NL2Repo-Bench</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>55.8</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.18)">42.3</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">36.2</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.15)">41.1</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.29)">47.6</td> |
| </tr> |
| <tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Reasoning, knowledge & instruction following</td></tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">GPQA-diamond</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.24)">89.4</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.22)">89.2</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">87.8</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.35)">90.3</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>91.3</strong></td> |
| </tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">HLE</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.32)">34.2</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.23)">30.8</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">24.0</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.33)">34.7</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>40.0</strong></td> |
| </tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">IFBench</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>82.0</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">79.5</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.19)">69.1</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.41)">79.1</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">62.5</td> |
| </tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">LiveBench overall</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.47)"><strong>76.9</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.42)">75.3</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">64.0</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;color:#9aa0a6">—</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.39)">74.5</td> |
| </tr> |
| <tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Vision</td></tr> |
| <tr> |
| <td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">MMMU-Pro</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;font-weight:600;background:rgba(42,120,214,0.10)">75.6</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.16)">76.3</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.47)"><strong>80</strong></td> |
| <td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);vertical-align:middle;font-size:14px;line-height:1.2;background:rgba(42,120,214,0.05)">75</td> |
| </tr> |
| </tbody></table> |
| <div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card unless noted; the Opus 4.6 AutomationBench score is from our own harness run and its JobBench score is a reported number, not our run; LiveBench rows use the public <a href="https://livebench.ai">livebench.ai</a> leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from <a href="https://artificialanalysis.ai">Artificial Analysis</a> (its Opus 4.6 MMMU-Pro entry runs at max effort); — = not reported.</div> |
| </div> |
|
|
| ### LiveBench category profile |
|
|
| *Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published [livebench.ai](https://livebench.ai) leaderboard entries (Smaug-Mini listed under the finetunes filter).* |
|
|
| <div align="center"> |
| <img src="figures/smaug_mini_livebench_categories.png" width="100%" alt="Smaug-Mini LiveBench category profile vs Qwen3.8-27B board entry"> |
| </div> |
|
|
| <details> |
| <summary><b>Notes</b></summary> |
|
|
| All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort `xhigh`, with generation budgets large enough that no score is truncation-bound. |
| |
| Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge. |
| |
| </details> |
| |
| ## 4. Training Approach |
| |
| Smaug-Mini was trained to make multi-turn tool use and automation episodes more reliable end to end. It is trained with on-policy reinforcement learning (GRPO) over multi-turn, tool-using automation episodes with verified, outcome-based rewards - an episode is rewarded for measurably completing its task in the environment. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta; the vision tower and multimodal projection are bitwise-identical to the base release. |
| |
| One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected. |
| |
| ## 5. Known Behaviors and Limitations |
| |
| Smaug-Mini redistributes deliberation rather than adding it: more reasoning per action but fewer actions per episode, finishing passing episodes about three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%. Refusals are zero for both models, and the general-capability benchmarks in §3 show the deliberation shift does not come at the cost of base skills. |
| |
| ## 6. Deployment |
| |
| Smaug-Mini serves as a drop-in replacement for Qwen3.8-27B on any stack that supports the base model. Reference vLLM invocation: |
| |
| ```bash |
| vllm serve abacusai/Smaug-Mini \ |
| --max-model-len 262144 \ |
| --reasoning-parser qwen3 \ |
| --enable-auto-tool-choice --tool-call-parser qwen3_coder |
| ``` |
| |
| Recommended sampling for agentic use: temperature 1.0, top_p 0.95, reasoning effort `xhigh` (the default). Recent vLLM versions return the chain-of-thought in the `reasoning` field of the message; when replaying conversation history, pass reasoning back under both `reasoning` and `reasoning_content` keys for portability across serving versions. Leave MTP-based speculative decoding off (§4). |
| |
| ## 7. License |
| |
| Apache 2.0, inherited from Qwen3.8-27B. |
| |
| ## 8. Citation |
| |
| ```bibtex |
| @misc{abacusai2026smaugmini, |
| title = {Smaug-Mini}, |
| author = {Abacus.AI}, |
| year = {2026}, |
| note = {Agentic fine-tune of Qwen/Qwen3.8-27B}, |
| url = {https://huggingface.co/abacusai/Smaug-Mini} |
| } |
| ``` |
| |
| The Smaug line and the DPO-Positive method behind it: |
| |
| ```bibtex |
| @article{pal2024smaug, |
| title={Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive}, |
| author={Pal, Arka and Karkhanis, Deep and Dooley, Samuel and |
| Roberts, Manley and Naidu, Siddartha and White, Colin}, |
| journal={arXiv preprint arXiv:2402.13228}, |
| year={2024} |
| } |
| ``` |
| |
| ## 9. Contact Us |
|
|
| If you have any questions, please reach out at [Abacus.AI](https://abacus.ai). |
|
|
