prann-ab commited on
Commit
3ba7a24
·
verified ·
1 Parent(s): a508ff5

Collapsible notes; lighten Training Approach; retitle and lighten Known Behaviors

Browse files
Files changed (1) hide show
  1. README.md +8 -17
README.md CHANGED
@@ -148,10 +148,15 @@ This card describes the training approach and the evaluation results. Dataset co
148
  <div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card; LiveBench rows use the public <a href="https://livebench.ai">livebench.ai</a> leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from <a href="https://artificialanalysis.ai">Artificial Analysis</a> (its Opus 4.6 MMMU-Pro entry runs at max effort); &mdash; = not reported.</div>
149
  </div>
150
 
 
 
 
151
  All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort `xhigh`, with generation budgets large enough that no score is truncation-bound.
152
 
153
  Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge.
154
 
 
 
155
  ### LiveBench category profile
156
 
157
  *Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published [livebench.ai](https://livebench.ai) leaderboard entries (Smaug-Mini listed under the finetunes filter).*
@@ -162,27 +167,13 @@ Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the
162
 
163
  ## 4. Training Approach
164
 
165
- Smaug-Mini follows the same central thesis as our other Smaug releases: **finetune in exactly the regime the model will be served in.** For this model that means multi-turn agentic episodes rendered in the model's native chat template — tool schemas, tool results, and interleaved reasoning present in context exactly as at inference time — with generation at the model's default (`xhigh`) reasoning effort throughout training, evaluation, and serving.
166
-
167
- Smaug-Mini is trained with on-policy reinforcement learning (GRPO) over multi-turn tool-using automation episodes with verified, outcome-based rewards: an episode is rewarded for measurably completing its task in the environment, not for matching a reference trajectory. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta after checkpoint selection. The vision tower and multimodal projection are bitwise-identical to the base release; the merged language trunk was verified to produce token-identical greedy output between the text-only and vision-restored assemblies.
168
 
169
  One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.
170
 
171
- ## 5. Behavioral Differences vs. the Base Model
172
-
173
- Measured over 600 AutomationBench episodes per model on identical serving, decode settings, and scaffold (base pooled over three full runs; reasoning measured from the same transcripts for both sides):
174
-
175
- | Per episode | Smaug-Mini | Qwen3.8-27B |
176
- |---|---|---|
177
- | Steps (p50 / p90) | **15 / 29** | 17 / 38 |
178
- | Tool calls (p50) | 32 | 34 |
179
- | Reasoning per action, chars (p50) | 2,004 | 1,722 |
180
- | Total reasoning, chars (p50 / p99) | 31.1k / 147k | 29.5k / 127k |
181
- | Episodes dying at the step ceiling | **1.0%** | 3.4% |
182
- | Steps to complete a passing episode (p50) | **14** | 17 |
183
- | Pass rate / average score | **38.3% / 0.77** | 31.8% / 0.68 |
184
 
185
- Smaug-Mini redistributes deliberation rather than adding it: roughly 16% more reasoning per action but 12–24% fewer actions per episode, finishing passing episodes three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%, and the average score rises more than the pass rate, meaning even failing episodes complete more of their assertions before stopping. Refusals are zero for both models, and the general-capability benchmarks in §3 (GPQA, LiveBench coding) show the deliberation shift does not come at the cost of base skills.
186
 
187
  ## 6. Deployment
188
 
 
148
  <div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best score in the row. Shading is relative within each row (darker = higher). Qwen3.6-27B, Qwen3.7-Plus, and Opus4.6 Max scores are as reported on the Qwen3.8-27B model card; LiveBench rows use the public <a href="https://livebench.ai">livebench.ai</a> leaderboard (its Opus 4.6 entry runs at high effort); the Qwen3.7-Plus AutomationBench score and all MMMU-Pro comparison scores are from <a href="https://artificialanalysis.ai">Artificial Analysis</a> (its Opus 4.6 MMMU-Pro entry runs at max effort); &mdash; = not reported.</div>
149
  </div>
150
 
151
+ <details>
152
+ <summary><b>Notes</b></summary>
153
+
154
  All Smaug-Mini numbers were produced at temperature 1.0, top_p 0.95, reasoning effort `xhigh`, with generation budgets large enough that no score is truncation-bound.
155
 
156
  Base points: JobBench, IFBench, and GPQA-diamond are the scores published on the Qwen3.8-27B model card; the AutomationBench base score is from our own run of the base model. JobBench was run under the official protocol with the OpenCode scaffold and an LLM judge.
157
 
158
+ </details>
159
+
160
  ### LiveBench category profile
161
 
162
  *Scores 0–100; overall = mean of the seven category averages. Both models' scores are the published [livebench.ai](https://livebench.ai) leaderboard entries (Smaug-Mini listed under the finetunes filter).*
 
167
 
168
  ## 4. Training Approach
169
 
170
+ Smaug-Mini was trained to make multi-turn tool use and automation episodes more reliable end to end. It is trained with on-policy reinforcement learning (GRPO) over multi-turn, tool-using automation episodes with verified, outcome-based rewards - an episode is rewarded for measurably completing its task in the environment. Training is a LoRA adapter over the language trunk only, merged into the base weights as a full delta; the vision tower and multimodal projection are bitwise-identical to the base release.
 
 
171
 
172
  One deployment note: the multi-token-prediction (MTP) head is inherited from the base model and was not retrained against the updated trunk. Speculative decoding via MTP should be left off; standard decoding is unaffected.
173
 
174
+ ## 5. Known Behaviors and Limitations
 
 
 
 
 
 
 
 
 
 
 
 
175
 
176
+ Smaug-Mini redistributes deliberation rather than adding it: more reasoning per action but fewer actions per episode, finishing passing episodes about three steps sooner at essentially unchanged total reasoning volume. The clearest shift is in failure behavior — episodes that burn the entire step budget without finishing drop from 3.4% to 1.0%. Refusals are zero for both models, and the general-capability benchmarks in §3 show the deliberation shift does not come at the cost of base skills.
177
 
178
  ## 6. Deployment
179