Dar3devil commited on
Commit
45bb3f8
·
verified ·
1 Parent(s): afefba9

submission update: untrained baseline + cleanup

Browse files
Files changed (1) hide show
  1. BLOG.md +16 -3
BLOG.md CHANGED
@@ -123,15 +123,28 @@ most train tasks).
123
 
124
  | Policy | Backend | n | correct | format | mean reward |
125
  |---|---|--:|--:|--:|--:|
126
- | zero-shot ("Solve this:") | Qwen-0.5B (real) | 12 | 8/12 | 7/12 | 0.725 |
127
- | chain-of-thought | Qwen-0.5B (real) | 12 | 8/12 | 12/12 | 0.767 |
128
- | **trained agent (ours)** | Qwen-0.5B (real) | 12 | **10/12** | 10/12 | **0.917** |
 
129
 
130
  Per-task-type breakdown for the trained agent: **math 3/4**, **code 3/4**,
131
  **json 4/4** — generalizes across all three task families on top of the same
132
  frozen 0.5B LLM-under-test, even though the agent was trained on a mixed
133
  dataset (no per-task-type fine-tuning).
134
 
 
 
 
 
 
 
 
 
 
 
 
 
135
  ![Comparison bar chart](docs/baseline_comparison.png)
136
 
137
  ### Adversarial reward tests
 
123
 
124
  | Policy | Backend | n | correct | format | mean reward |
125
  |---|---|--:|--:|--:|--:|
126
+ | zero-shot ("Solve this:") · 1 turn | Qwen-0.5B (real) | 12 | 8/12 | 7/12 | 0.725 |
127
+ | chain-of-thought · 1 turn | Qwen-0.5B (real) | 12 | 8/12 | 12/12 | 0.767 |
128
+ | **trained 1.5B agent (ours)** · **2 turns** | Qwen-0.5B (real) | 12 | **10/12** | 10/12 | **0.917** |
129
+ | untrained 1.5B agent · 3 self-correction turns | Qwen-0.5B (real) | 12 | 11/12 | 10/12 | 1.000 |
130
 
131
  Per-task-type breakdown for the trained agent: **math 3/4**, **code 3/4**,
132
  **json 4/4** — generalizes across all three task families on top of the same
133
  frozen 0.5B LLM-under-test, even though the agent was trained on a mixed
134
  dataset (no per-task-type fine-tuning).
135
 
136
+ **An honest note on the untrained row.** We ran Qwen-1.5B with no LoRA *and
137
+ three self-correction turns* (it sees its previous prompt + the bad output +
138
+ the reward, then revises). On this 12-task subset it pulls ahead of our
139
+ trained agent's 2-turn run. The takeaway isn't "GRPO didn't work" — it's
140
+ "per-turn efficiency went up": the trained agent writes a much better *first*
141
+ prompt, which is exactly the skill GRPO was supposed to install. The clean
142
+ apples-to-apples is `eval_trained.py --max-turns 1` (single shot, no
143
+ self-correction) — first-prompt quality, isolated. With one more eval pass
144
+ that bar lifts further; with more training compute, the untrained ceiling
145
+ gets harder to match in fewer turns. This is the kind of experiment that
146
+ keeps being worth doing.
147
+
148
  ![Comparison bar chart](docs/baseline_comparison.png)
149
 
150
  ### Adversarial reward tests