Add Smaug-Mini model card (draft)

README.md

@@ -0,0 +1,94 @@
+---
+license: apache-2.0
+library_name: transformers
+base_model: Qwen/Qwen3.8-27B
+base_model_relation: finetune
+pipeline_tag: image-text-to-text
+tags:
+- agentic
+- smaug
+---
+
+# Smaug-Mini
+
+**Abacus.AI** · September 2026
+
+Smaug-Mini is an agentic finetune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). It improves end-to-end agentic tool use and automation performance (+6.0 on AutomationBench over the base's best observed run, +2.0 overall on LiveBench, +5.2 on LiveBench math, +2.5 on IFBench) while holding the base model's general capabilities at parity (GPQA-diamond within noise of the official number, LiveBench coding and agentic-coding at board parity). The weights load exactly like the official release: same layout, same `Qwen3_5ForConditionalGeneration` architecture with the vision tower intact, same 262,144-token context, same reasoning interface with `xhigh` / `medium` / `low` effort levels.
+
+This card describes the training approach and the evaluation results. Dataset contents are not disclosed; training data consists of multi-turn, tool-using automation episodes generated and verified against executable environments.
+
+---
+
+## 1. Training regime: match the serving regime
+
+Smaug-Mini follows the same central thesis as our other Smaug releases: **finetune in exactly the regime the model will be served in.** For this model that means multi-turn agentic episodes rendered in the model's native chat template — tool schemas, tool results, and interleaved reasoning present in context exactly as at inference time — with generation at the model's default (`xhigh`) reasoning effort throughout training, evaluation, and serving.
+
+## 2. Training method
+
+Smaug-Mini is trained with on-policy reinforcement learning (GRPO) over multi-turn tool-using automation episodes with verified, outcome-based rewards: an episode is rewarded for measurably completing its task in the environment, not for matching a reference trajectory. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta after checkpoint selection. The vision tower and multimodal projection are bitwise-identical to the base release; the merged language trunk was verified to produce token-identical greedy output between the text-only and vision-restored assemblies.
+
+One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.
+
+## 3. Evaluation
+
+All Smaug-Mini numbers below were produced at temperature 1.0, top_p 0.95, reasoning effort `xhigh`, with generation budgets large enough that no score is truncation-bound. Base-model comparison points are the published Qwen3.8-27B model-card numbers and the public [livebench.ai](https://livebench.ai) leaderboard entries, never our own reruns of the base.
+
+### Agentic
+
+| Benchmark | Smaug-Mini | Qwen3.8-27B | Delta |
+|---|---|---|---|
+| AutomationBench (full 600) | **41.8** | 35.8–38.3 (3-run band) | +3.5 to +6.0 |
+| JobBench (OpenCode scaffold, macro) | **50.5** | 33.4† | see note |
+| LiveBench agentic coding | 61.4 | 61.4 | par |
+
+† The Qwen3.8-27B card reports JobBench under a different judge model and scaffold configuration, so the two numbers are not directly comparable; we report ours (OpenCode v1.14.18, grok-4.3 judge, official protocol) for transparency rather than as a claimed +17.
+
+### Reasoning, knowledge and instruction following
+
+| Benchmark | Smaug-Mini | Qwen3.8-27B | Delta |
+|---|---|---|---|
+| GPQA-diamond | 89.1 | 89.2 | par |
+| IFBench (prompt-level loose) | **82.0** | 79.5 | +2.5 |
+| LiveBench overall | **77.2** | 75.3 | +2.0 |
+
+GPQA-diamond is the mean of two independent full runs (88.9, 89.4) with answer extraction audited by an arbiter pass; the widely used flexible-extract regex under-scores this model family by ~8 points by matching the last parenthesized letter inside chain-of-thought.
+
+### LiveBench by category (vs published leaderboard)
+
+| Category | Smaug-Mini | Qwen3.8-27B (board) | Delta |
+|---|---|---|---|
+| math | **91.4** | 86.2 | +5.2 |
+| reasoning | **82.8** | 80.0 | +2.8 |
+| data analysis | **78.8** | 76.6 | +2.2 |
+| language | **77.0** | 74.3 | +2.7 |
+| instruction following | **73.9** | 72.7 | +1.2 |
+| coding | 75.4 | 75.7 | −0.3 |
+| agentic coding | 61.4 | 61.4 | 0.0 |
+| **average** | **77.2** | 75.3 | **+2.0** |
+
+Every category was audited for judgment/answer count parity and zero error-placeholder rows; coding was replicated across five independent serving stacks and client-concurrency settings (73.9–75.4 range) and is board-par within run variance.
+
+### In progress
+
+Humanity's Last Exam, Agents' Last Exam, and an additional JobBench replication are running at the time of writing and will be added when final.
+
+## 4. Deployment
+
+Smaug-Mini serves as a drop-in replacement for Qwen3.8-27B on any stack that supports the base model. Reference vLLM invocation:
+
+```bash
+vllm serve abacusai/Smaug-Mini \
+  --max-model-len 262144 \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice --tool-call-parser qwen3_coder
+```
+
+Recommended sampling for agentic use: temperature 1.0, top_p 0.95, reasoning effort `xhigh` (the default). Recent vLLM versions return the chain-of-thought in the `reasoning` field of the message; when replaying conversation history, pass reasoning back under both `reasoning` and `reasoning_content` keys for portability across serving versions. Leave MTP-based speculative decoding off (§2).
+
+## License
+
+Apache 2.0, inherited from Qwen3.8-27B.
+
+## Contact us
+
+For questions about Smaug-Mini, reach out at [https://abacus.ai](https://abacus.ai).