Spaces:
Sleeping
Sleeping
submission update: untrained baseline + cleanup
Browse files
BLOG.md
CHANGED
|
@@ -123,15 +123,28 @@ most train tasks).
|
|
| 123 |
|
| 124 |
| Policy | Backend | n | correct | format | mean reward |
|
| 125 |
|---|---|--:|--:|--:|--:|
|
| 126 |
-
| zero-shot ("Solve this:") | Qwen-0.5B (real) | 12 | 8/12 | 7/12 | 0.725 |
|
| 127 |
-
| chain-of-thought | Qwen-0.5B (real) | 12 | 8/12 | 12/12 | 0.767 |
|
| 128 |
-
| **trained agent (ours)** | Qwen-0.5B (real) | 12 | **10/12** | 10/12 | **0.917** |
|
|
|
|
| 129 |
|
| 130 |
Per-task-type breakdown for the trained agent: **math 3/4**, **code 3/4**,
|
| 131 |
**json 4/4** — generalizes across all three task families on top of the same
|
| 132 |
frozen 0.5B LLM-under-test, even though the agent was trained on a mixed
|
| 133 |
dataset (no per-task-type fine-tuning).
|
| 134 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 135 |

|
| 136 |
|
| 137 |
### Adversarial reward tests
|
|
|
|
| 123 |
|
| 124 |
| Policy | Backend | n | correct | format | mean reward |
|
| 125 |
|---|---|--:|--:|--:|--:|
|
| 126 |
+
| zero-shot ("Solve this:") · 1 turn | Qwen-0.5B (real) | 12 | 8/12 | 7/12 | 0.725 |
|
| 127 |
+
| chain-of-thought · 1 turn | Qwen-0.5B (real) | 12 | 8/12 | 12/12 | 0.767 |
|
| 128 |
+
| **trained 1.5B agent (ours)** · **2 turns** | Qwen-0.5B (real) | 12 | **10/12** | 10/12 | **0.917** |
|
| 129 |
+
| untrained 1.5B agent · 3 self-correction turns | Qwen-0.5B (real) | 12 | 11/12 | 10/12 | 1.000 |
|
| 130 |
|
| 131 |
Per-task-type breakdown for the trained agent: **math 3/4**, **code 3/4**,
|
| 132 |
**json 4/4** — generalizes across all three task families on top of the same
|
| 133 |
frozen 0.5B LLM-under-test, even though the agent was trained on a mixed
|
| 134 |
dataset (no per-task-type fine-tuning).
|
| 135 |
|
| 136 |
+
**An honest note on the untrained row.** We ran Qwen-1.5B with no LoRA *and
|
| 137 |
+
three self-correction turns* (it sees its previous prompt + the bad output +
|
| 138 |
+
the reward, then revises). On this 12-task subset it pulls ahead of our
|
| 139 |
+
trained agent's 2-turn run. The takeaway isn't "GRPO didn't work" — it's
|
| 140 |
+
"per-turn efficiency went up": the trained agent writes a much better *first*
|
| 141 |
+
prompt, which is exactly the skill GRPO was supposed to install. The clean
|
| 142 |
+
apples-to-apples is `eval_trained.py --max-turns 1` (single shot, no
|
| 143 |
+
self-correction) — first-prompt quality, isolated. With one more eval pass
|
| 144 |
+
that bar lifts further; with more training compute, the untrained ceiling
|
| 145 |
+
gets harder to match in fewer turns. This is the kind of experiment that
|
| 146 |
+
keeps being worth doing.
|
| 147 |
+
|
| 148 |

|
| 149 |
|
| 150 |
### Adversarial reward tests
|