rtferraz commited on
Commit
f46c90b
·
verified ·
1 Parent(s): b27b03d

add: FAL demo results report (preliminary — eval crash pending fix)

Browse files
Files changed (1) hide show
  1. docs/reports/fal_demo_results.md +139 -0
docs/reports/fal_demo_results.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Future-as-Label Demo: Results Report
2
+
3
+ **Date:** 2026-05-11
4
+ **Status:** Training complete ✅, Evaluation partially complete (crash in per-task ECE computation)
5
+ **Runtime:** 79 minutes on Tesla T4, ~$0.80 total cost
6
+
7
+ ---
8
+
9
+ ## 1. Training Summary
10
+
11
+ | Phase | Steps | Reward | What happened |
12
+ |---|---|---|---|
13
+ | 0–11 | Warmup | 0.08 | 8% parse rate, model can't produce "PROB:" format |
14
+ | 11–65 | Format learning | 0.08→0.30 | Breakthrough — model discovers parseable format |
15
+ | 65–100 | Format consolidation | 0.30→0.70 | 100% parse rate achieved, frac_zero_std drops to 0 |
16
+ | 100–800 | Calibration refinement | 0.65–0.83 | Stable plateau, model refining probability estimates |
17
+
18
+ **Final training metrics (step 800):**
19
+ - reward: 0.62–0.75 (oscillating, healthy)
20
+ - frac_reward_zero_std: 0 (every batch has signal)
21
+ - entropy: 1.6–2.0 (stable, no collapse)
22
+ - clipped_ratio: 0.5–0.8 (model learned shorter outputs)
23
+ - completions/mean_length: decreasing from 48→32 tokens (learning efficiency)
24
+ - epoch: 0.53 (seen ~half the data once)
25
+ - train_loss: 0.0011
26
+
27
+ ---
28
+
29
+ ## 2. Evaluation Results (Partial)
30
+
31
+ ### Parse Rate — The Headline Finding
32
+
33
+ | Model | Parse Rate | Meaning |
34
+ |---|---|---|
35
+ | **Base** | **0/134 (0%)** | Base model never outputs "PROB: X.XX" — completely ignores format instruction |
36
+ | **Tuned** | **134/134 (100%)** | GRPO taught perfect format compliance from zero |
37
+
38
+ This is the single most important result: **GRPO took a model with 0% format compliance and achieved 100% in 800 steps (79 minutes, $0.80).** This replicates the V4.2 finding (extraction JSON format learning) in a completely different domain.
39
+
40
+ ### Brier Score (Overall)
41
+
42
+ | Model | Brier ↓ | Interpretation |
43
+ |---|---|---|
44
+ | Base | 0.2500 | Always predicts 0.5 (fallback for unparseable) → (0.5−y)² on balanced data = 0.25 |
45
+ | Tuned | 0.2905 | Produces real probabilities but slightly miscalibrated |
46
+
47
+ **Important context:** The base "0.25" is not a real prediction — it's the artifact of 0% parse rate falling back to 0.5 on a balanced (50/50) dataset. The fair comparison is tuned model vs. **base rate predictor** (always predict the marginal probability per task).
48
+
49
+ ### ECE (Overall)
50
+
51
+ | Model | ECE ↓ |
52
+ |---|---|
53
+ | Base | 0.0075 | Meaningless — all predictions are exactly 0.5, so trivially "calibrated" to that bin |
54
+ | Tuned | 0.1740 | Model spreads predictions across bins but not perfectly calibrated |
55
+
56
+ ---
57
+
58
+ ## 3. Interpretation
59
+
60
+ ### What the model learned
61
+
62
+ 1. **Format (complete success):** The model went from never producing "PROB:" to always producing it. This confirms V4.2's finding: GRPO can teach output format through reward shaping alone.
63
+
64
+ 2. **Probability values (partial):** The model produces different probabilities for different inputs (reward_std > 0 throughout), meaning it's not just outputting a constant. But the Brier score (0.29) being worse than naive 0.5 means its probabilities are miscalibrated — likely overconfident on wrong predictions.
65
+
66
+ 3. **No entropy collapse:** Unlike V4.2 (which collapsed at 1.5 epochs), this run completed 0.53 epochs without collapse. The simpler output format (just a number) and shorter completions prevent the diversity loss that killed V4.2.
67
+
68
+ ### Why calibration is worse than naive baseline
69
+
70
+ The tuned Brier (0.29) > base Brier (0.25) doesn't mean the model is "worse" — it means:
71
+
72
+ - Base always predicts 0.5 → on balanced data, this is optimal in expectation (minimizes expected Brier)
73
+ - Tuned model produces varied predictions (0.2, 0.7, 0.8, etc.) but isn't perfectly calibrated
74
+ - A model that says "0.8" when the true rate is 0.6 gets penalized more than one that says "0.5"
75
+
76
+ This is the **overconfidence problem** the Future-as-Label paper addresses with more training (160 steps on 32B vs our 800 steps on 0.5B). With G=2 (minimal rollouts), the calibration signal is weak — the model learned format but doesn't have enough reward variance to learn precise calibration.
77
+
78
+ ### Comparison to paper results
79
+
80
+ | | Paper (32B, 160 steps) | Our demo (0.5B, 800 steps) |
81
+ |---|---|---|
82
+ | Format compliance | 100% (from high baseline) | **0% → 100%** (from zero) |
83
+ | Brier improvement | 0.247 → 0.179 (−27%) | 0.25 → 0.29 (+16%) — miscalibrated |
84
+ | ECE improvement | 0.218 → 0.104 (−52%) | N/A → 0.174 |
85
+ | G (rollouts) | 4 | 2 (minimum viable) |
86
+ | Model capacity | 32B | 0.5B |
87
+
88
+ **Expected:** A 0.5B model with G=2 was always going to be limited on calibration — the research question was "does it learn anything?" and the answer is definitively yes (format learning), partially (probability production), but not yet (calibration).
89
+
90
+ ---
91
+
92
+ ## 4. What Would Improve Results
93
+
94
+ Based on V4.2 lessons + paper methodology:
95
+
96
+ 1. **G=8 or G=16** (requires larger GPU or batch adjustment): More rollouts per prompt = more reward variance = stronger calibration signal
97
+ 2. **More steps** (1500-3000): The model is at 0.53 epochs, still learning. V4.2 peaked at ~1.5 epochs.
98
+ 3. **Larger model** (1.5B-3B): The paper's 32B result shows calibration improves with scale
99
+ 4. **Log score reward** (paper's actual method): Harsher penalty for overconfidence may push calibration faster
100
+
101
+ ---
102
+
103
+ ## 5. Technical Findings
104
+
105
+ ### Training was remarkably stable
106
+
107
+ - No entropy collapse (entropy stayed 1.5-2.0 throughout)
108
+ - No reward hacking (reward oscillated naturally, never spiked to 1.0)
109
+ - Completion length decreased organically (48→32 tokens) as model learned to be concise
110
+ - Step time decreased (6.2s→5.5s) as completions got shorter
111
+
112
+ ### G=2 provides enough signal for format learning, not for calibration
113
+
114
+ - Format learning happened in ~80 steps (0→100% parse rate)
115
+ - Calibration refinement showed slow improvement (0.08→0.70 reward) but didn't translate to better Brier on eval
116
+ - The frac_zero_std was 0 after step 100, meaning GRPO always had signal — but with only 2 rollouts, the signal is noisy
117
+
118
+ ### T4 is sufficient for 0.5B GRPO without mixed precision
119
+
120
+ - Peak VRAM: ~4GB (model) + ~2GB (generations) ≈ 6GB of 16GB used
121
+ - No OOM, no bf16 issues after disabling mixed precision
122
+ - Cost: $0.80 for full 800-step run — viable for rapid iteration
123
+
124
+ ---
125
+
126
+ ## 6. Conclusion
127
+
128
+ **The Future-as-Label PoC succeeded on format learning and failed on calibration.** This is the "format learned but no calibration" outcome predicted in ADR-003 — it demonstrates that GRPO can teach a 0.5B model to output probabilities (from 0% → 100% format compliance), but the model doesn't have sufficient capacity + signal to learn *well-calibrated* probabilities with only G=2 rollouts.
129
+
130
+ The result is scientifically informative: it establishes a lower bound on what's needed for calibration learning, and confirms that format-learning generalizes across tasks (JSON in V4.2, probabilities here).
131
+
132
+ ---
133
+
134
+ ## 7. Pending
135
+
136
+ - [ ] Fix eval crash (ZeroDivisionError in per-task ECE when a task has no base predictions in a bin)
137
+ - [ ] Re-run eval only (no retraining needed — model saved to volume)
138
+ - [ ] Add per-task breakdown to determine which tasks the model calibrates best on
139
+ - [ ] Compare tuned Brier to base-rate Brier (always predict marginal) — the true baseline