rtferraz commited on
Commit
99c09f5
Β·
verified Β·
1 Parent(s): 3fc0eb8

update: FAL results report with final eval numbers + conclusions

Browse files
Files changed (1) hide show
  1. docs/reports/fal_demo_results.md +102 -64
docs/reports/fal_demo_results.md CHANGED
@@ -1,8 +1,8 @@
1
- # Future-as-Label Demo: Results Report
2
 
3
  **Date:** 2026-05-11
4
- **Status:** Training complete βœ…, Evaluation partially complete (crash in per-task ECE computation)
5
- **Runtime:** 79 minutes on Tesla T4, ~$0.80 total cost
6
 
7
  ---
8
 
@@ -26,32 +26,36 @@
26
 
27
  ---
28
 
29
- ## 2. Evaluation Results (Partial)
30
 
31
- ### Parse Rate β€” The Headline Finding
32
 
33
  | Model | Parse Rate | Meaning |
34
  |---|---|---|
35
  | **Base** | **0/134 (0%)** | Base model never outputs "PROB: X.XX" β€” completely ignores format instruction |
36
  | **Tuned** | **134/134 (100%)** | GRPO taught perfect format compliance from zero |
37
 
38
- This is the single most important result: **GRPO took a model with 0% format compliance and achieved 100% in 800 steps (79 minutes, $0.80).** This replicates the V4.2 finding (extraction JSON format learning) in a completely different domain.
39
 
40
- ### Brier Score (Overall)
41
 
42
- | Model | Brier ↓ | Interpretation |
43
- |---|---|---|
44
- | Base | 0.2500 | Always predicts 0.5 (fallback for unparseable) β†’ (0.5βˆ’y)Β² on balanced data = 0.25 |
45
- | Tuned | 0.2905 | Produces real probabilities but slightly miscalibrated |
 
 
 
46
 
47
- **Important context:** The base "0.25" is not a real prediction β€” it's the artifact of 0% parse rate falling back to 0.5 on a balanced (50/50) dataset. The fair comparison is tuned model vs. **base rate predictor** (always predict the marginal probability per task).
48
 
49
- ### ECE (Overall)
 
 
 
 
50
 
51
- | Model | ECE ↓ |
52
- |---|---|
53
- | Base | 0.0075 | Meaningless β€” all predictions are exactly 0.5, so trivially "calibrated" to that bin |
54
- | Tuned | 0.1740 | Model spreads predictions across bins but not perfectly calibrated |
55
 
56
  ---
57
 
@@ -59,81 +63,115 @@ This is the single most important result: **GRPO took a model with 0% format com
59
 
60
  ### What the model learned
61
 
62
- 1. **Format (complete success):** The model went from never producing "PROB:" to always producing it. This confirms V4.2's finding: GRPO can teach output format through reward shaping alone.
63
 
64
- 2. **Probability values (partial):** The model produces different probabilities for different inputs (reward_std > 0 throughout), meaning it's not just outputting a constant. But the Brier score (0.29) being worse than naive 0.5 means its probabilities are miscalibrated β€” likely overconfident on wrong predictions.
65
 
66
- 3. **No entropy collapse:** Unlike V4.2 (which collapsed at 1.5 epochs), this run completed 0.53 epochs without collapse. The simpler output format (just a number) and shorter completions prevent the diversity loss that killed V4.2.
67
 
68
- ### Why calibration is worse than naive baseline
69
 
70
- The tuned Brier (0.29) > base Brier (0.25) doesn't mean the model is "worse" β€” it means:
 
 
71
 
72
- - Base always predicts 0.5 β†’ on balanced data, this is optimal in expectation (minimizes expected Brier)
73
- - Tuned model produces varied predictions (0.2, 0.7, 0.8, etc.) but isn't perfectly calibrated
74
- - A model that says "0.8" when the true rate is 0.6 gets penalized more than one that says "0.5"
75
 
76
- This is the **overconfidence problem** the Future-as-Label paper addresses with more training (160 steps on 32B vs our 800 steps on 0.5B). With G=2 (minimal rollouts), the calibration signal is weak β€” the model learned format but doesn't have enough reward variance to learn precise calibration.
77
 
78
- ### Comparison to paper results
79
 
80
- | | Paper (32B, 160 steps) | Our demo (0.5B, 800 steps) |
81
- |---|---|---|
82
- | Format compliance | 100% (from high baseline) | **0% β†’ 100%** (from zero) |
83
- | Brier improvement | 0.247 β†’ 0.179 (βˆ’27%) | 0.25 β†’ 0.29 (+16%) β€” miscalibrated |
84
- | ECE improvement | 0.218 β†’ 0.104 (βˆ’52%) | N/A β†’ 0.174 |
85
- | G (rollouts) | 4 | 2 (minimum viable) |
86
- | Model capacity | 32B | 0.5B |
87
 
88
- **Expected:** A 0.5B model with G=2 was always going to be limited on calibration β€” the research question was "does it learn anything?" and the answer is definitively yes (format learning), partially (probability production), but not yet (calibration).
89
 
90
- ---
 
 
 
 
 
 
 
91
 
92
- ## 4. What Would Improve Results
93
 
94
- Based on V4.2 lessons + paper methodology:
 
 
95
 
96
- 1. **G=8 or G=16** (requires larger GPU or batch adjustment): More rollouts per prompt = more reward variance = stronger calibration signal
97
- 2. **More steps** (1500-3000): The model is at 0.53 epochs, still learning. V4.2 peaked at ~1.5 epochs.
98
- 3. **Larger model** (1.5B-3B): The paper's 32B result shows calibration improves with scale
99
- 4. **Log score reward** (paper's actual method): Harsher penalty for overconfidence may push calibration faster
100
 
101
  ---
102
 
103
- ## 5. Technical Findings
104
 
105
- ### Training was remarkably stable
 
 
 
 
 
 
 
106
 
107
- - No entropy collapse (entropy stayed 1.5-2.0 throughout)
108
- - No reward hacking (reward oscillated naturally, never spiked to 1.0)
109
- - Completion length decreased organically (48β†’32 tokens) as model learned to be concise
110
- - Step time decreased (6.2s→5.5s) as completions got shorter
111
 
112
- ### G=2 provides enough signal for format learning, not for calibration
113
 
114
- - Format learning happened in ~80 steps (0β†’100% parse rate)
115
- - Calibration refinement showed slow improvement (0.08β†’0.70 reward) but didn't translate to better Brier on eval
116
- - The frac_zero_std was 0 after step 100, meaning GRPO always had signal β€” but with only 2 rollouts, the signal is noisy
117
 
118
- ### T4 is sufficient for 0.5B GRPO without mixed precision
119
 
120
- - Peak VRAM: ~4GB (model) + ~2GB (generations) β‰ˆ 6GB of 16GB used
121
- - No OOM, no bf16 issues after disabling mixed precision
122
- - Cost: $0.80 for full 800-step run β€” viable for rapid iteration
 
 
 
 
 
 
 
 
 
123
 
124
  ---
125
 
126
- ## 6. Conclusion
 
 
 
 
 
 
 
127
 
128
- **The Future-as-Label PoC succeeded on format learning and failed on calibration.** This is the "format learned but no calibration" outcome predicted in ADR-003 β€” it demonstrates that GRPO can teach a 0.5B model to output probabilities (from 0% β†’ 100% format compliance), but the model doesn't have sufficient capacity + signal to learn *well-calibrated* probabilities with only G=2 rollouts.
129
 
130
- The result is scientifically informative: it establishes a lower bound on what's needed for calibration learning, and confirms that format-learning generalizes across tasks (JSON in V4.2, probabilities here).
 
 
 
 
 
 
 
 
 
 
 
 
131
 
132
  ---
133
 
134
- ## 7. Pending
135
 
136
- - [ ] Fix eval crash (ZeroDivisionError in per-task ECE when a task has no base predictions in a bin)
137
- - [ ] Re-run eval only (no retraining needed β€” model saved to volume)
138
- - [ ] Add per-task breakdown to determine which tasks the model calibrates best on
139
- - [ ] Compare tuned Brier to base-rate Brier (always predict marginal) β€” the true baseline
 
 
 
 
 
1
+ # Future-as-Label Demo: Final Results Report
2
 
3
  **Date:** 2026-05-11
4
+ **Status:** Complete βœ… (training + evaluation)
5
+ **Runtime:** 79 min training + 5 min eval on Tesla T4, ~$0.85 total cost
6
 
7
  ---
8
 
 
26
 
27
  ---
28
 
29
+ ## 2. Final Evaluation Results
30
 
31
+ ### Parse Rate β€” The Headline Result
32
 
33
  | Model | Parse Rate | Meaning |
34
  |---|---|---|
35
  | **Base** | **0/134 (0%)** | Base model never outputs "PROB: X.XX" β€” completely ignores format instruction |
36
  | **Tuned** | **134/134 (100%)** | GRPO taught perfect format compliance from zero |
37
 
38
+ **GRPO took a model with 0% format compliance and achieved 100% in 800 steps (79 minutes, $0.85).**
39
 
40
+ ### Overall Metrics
41
 
42
+ | Metric | Base | Tuned | Ξ” | Interpretation |
43
+ |---|---|---|---|---|
44
+ | **Brier ↓** | 0.2500 | 0.2653 | +0.0153 | Tuned slightly worse (but base is artificial 0.5 fallback) |
45
+ | **ECE ↓** | 0.0075 | 0.1394 | +0.1319 | Tuned spreads predictions across range; not perfectly calibrated |
46
+ | **Base rate Brier** | 0.2499 | β€” | β€” | Always-predict-base-rate baseline |
47
+ | **Base rate** | 0.4925 | β€” | β€” | ~49% positive rate (balanced) |
48
+ | **Wilcoxon p** | β€” | 0.9495 | β€” | ❌ Not significant |
49
 
50
+ ### Per-Task Breakdown
51
 
52
+ | Task | Base Brier | Tuned Brier | Ξ” | N |
53
+ |---|---|---|---|---|
54
+ | repurchase_90d | N/A | N/A | β€” | 0 (no eval samples generated for this task) |
55
+ | review_positive | 0.2500 | 0.2723 | +0.0223 | 67 |
56
+ | delivery_ontime | 0.2500 | 0.2583 | +0.0083 | 67 |
57
 
58
+ **Note:** repurchase_90d has 0 eval samples β€” a bug in the data pipeline's temporal censoring filter (the 90-day window extends past the eval end date, filtering out all eval candidates). This means only 2 of 3 tasks were evaluated.
 
 
 
59
 
60
  ---
61
 
 
63
 
64
  ### What the model learned
65
 
66
+ 1. **Format (complete success):** 0% β†’ 100% parse rate. The model went from never producing "PROB:" to always producing it. Confirms V4.2's finding: GRPO teaches format through reward alone.
67
 
68
+ 2. **Probability discrimination (partial):** The model produces varied probabilities (not a constant), as evidenced by reward_std > 0 and ECE > 0. It IS using the input features to modulate its predictions.
69
 
70
+ 3. **Calibration (not achieved):** Tuned Brier (0.265) is slightly worse than the base fallback (0.250). The model's probability estimates don't yet align with true event frequencies.
71
 
72
+ ### Why the base "wins" on Brier
73
 
74
+ The base Brier of 0.25 is **not a real prediction** β€” it's the mathematical result of:
75
+ - 0% parse rate β†’ all predictions fall back to 0.5
76
+ - On balanced data (base_rate=0.49): (0.5 - y)Β² averages to 0.25
77
 
78
+ This is the **optimal constant predictor** for balanced binary data. Any model that actually tries to differentiate (predict 0.8 for some, 0.2 for others) will score worse unless its predictions are well-calibrated. The tuned model at Brier=0.265 is only +0.015 above this floor β€” meaning it's approximately correct on average but slightly overconfident on incorrect predictions.
 
 
79
 
80
+ ### The delivery_ontime task nearly matches baseline
81
 
82
+ Delivery task Brier: 0.2583 (only +0.008 above 0.25). This is very close to the optimal constant β€” the model may have learned to predict near-0.5 for this task (appropriate for balanced data) while spreading predictions for review_positive (0.2723, slightly worse).
83
 
84
+ ---
 
 
 
 
 
 
85
 
86
+ ## 4. Comparison to Paper & V4.2
87
 
88
+ | Dimension | Paper (32B) | V4.2 Commerce | FAL Demo (0.5B) |
89
+ |---|---|---|---|
90
+ | Format learning | N/A (already formatted) | +29.5% extraction | **0%β†’100% parse** |
91
+ | Calibration | βˆ’27% Brier | N/A | +6% Brier (slightly worse) |
92
+ | Model | Qwen3-32B | Tucano2-0.5B | Tucano2-0.5B |
93
+ | G (rollouts) | 4 | 16 | 2 |
94
+ | Training steps | 160 | 1,100 (best) | 800 |
95
+ | Cost | ~$100+ (est) | ~$18 | **$0.85** |
96
 
97
+ ### Key insight: format learning is the consistent GRPO win at 0.5B
98
 
99
+ Both V4.2 and FAL demo show the same pattern:
100
+ - **Format/structure learning:** massive improvement (JSON types, probability format)
101
+ - **Content/calibration quality:** marginal or no improvement at 0.5B
102
 
103
+ GRPO at this scale is fundamentally a **formatting tool**, not a reasoning tool.
 
 
 
104
 
105
  ---
106
 
107
+ ## 5. What Would Improve Calibration
108
 
109
+ | Change | Expected impact | Evidence |
110
+ |---|---|---|
111
+ | G=8+ (more rollouts) | Strong | Paper uses G=4; our G=2 gives 70% zero-std early on |
112
+ | 1.5B-3B model | Strong | Paper used 32B; 0.5B may lack capacity for probability reasoning |
113
+ | Log score reward | Moderate | Harsher penalty for overconfidence (paper's actual method) |
114
+ | More training (1500+ steps) | Moderate | Only 0.53 epochs completed; V4.2 peaked at 1.5 |
115
+ | Fix repurchase task in eval | Minor | Adds 67 more eval samples for significance |
116
+ | L40S GPU + G=16 batch=16 | Strong | Matches V4.2 signal strength |
117
 
118
+ ---
 
 
 
119
 
120
+ ## 6. Technical Findings
121
 
122
+ ### Data pipeline bug: repurchase_90d has 0 eval samples
 
 
123
 
124
+ The temporal censoring filter (`order_purchase_timestamp <= dataset_end - 90 days`) combined with the eval window (2018-06-01 to 2018-08-31) leaves zero observable repurchase outcomes β€” because you need 90 days AFTER the order to observe whether repurchase happened, but the dataset ends before that window closes for eval-period orders.
125
 
126
+ **Fix for next run:** Either extend the eval end date or shorten the repurchase window to 30 days for the eval split.
127
+
128
+ ### Training was remarkably stable
129
+
130
+ - No entropy collapse (entropy 1.5-2.0 throughout all 800 steps)
131
+ - No reward hacking
132
+ - Completion length decreased organically (48β†’32 tokens)
133
+ - No gradient explosion (grad_norm 0-2.5, healthy range)
134
+
135
+ ### The "trivially calibrated" base model problem
136
+
137
+ When base parse rate = 0%, evaluating calibration is meaningless β€” the base model isn't making predictions, it's just outputting text. The only fair comparison would be against a **prompted** base model that CAN produce probabilities (e.g., with few-shot examples). This wasn't implemented.
138
 
139
  ---
140
 
141
+ ## 7. Conclusions
142
+
143
+ ### βœ… Success: Format learning generalizes
144
+
145
+ GRPO + reward shaping teaches format compliance on any output structure:
146
+ - V4.2: JSON with correct types (+29.5%)
147
+ - FAL: Probability decimal format (0% β†’ 100%)
148
+ - Both: ~80-100 steps to learn, ~$1 cost on T4
149
 
150
+ ### ⚠️ Partial: Probability production without calibration
151
 
152
+ The model produces varied probabilities (not a constant) β€” it IS differentiating between inputs. But the predictions aren't calibrated enough to beat the naive 0.5 baseline. This is the expected outcome for G=2 on 0.5B: **necessary but not sufficient signal for calibration learning**.
153
+
154
+ ### ❌ Not achieved: Brier improvement over baseline
155
+
156
+ The success criterion "tuned Brier < base Brier" was not met. The Wilcoxon test confirms: p=0.95, no significant difference. This is the "0.5B below minimum viable scale for calibration" outcome from ADR-003.
157
+
158
+ ### The experiment is still informative
159
+
160
+ We now know:
161
+ 1. **Lower bound for calibration:** 0.5B with G=2 is below the threshold
162
+ 2. **Format learning works universally:** Another domain, same result as V4.2
163
+ 3. **Stability is not an issue:** 800 steps without collapse (vs. V4.2's collapse at step 1100)
164
+ 4. **Cost:** $0.85 for a complete GRPO experiment including eval β€” this workflow is cheap enough for daily iteration
165
 
166
  ---
167
 
168
+ ## 8. Files
169
 
170
+ | File | Contents |
171
+ |---|---|
172
+ | Model checkpoint | Modal volume `fal-checkpoints` β†’ `/checkpoints/fal-grpo/final/` |
173
+ | Eval data | Modal volume β†’ `/checkpoints/fal-grpo/eval_dataset.json` |
174
+ | Results JSON | Modal volume β†’ `/checkpoints/fal-grpo/fal_results.json` |
175
+ | Training script | `fal_demo/train_modal.py` |
176
+ | ADR | `docs/ADR-003-future-as-label.md` |
177
+ | Deployment postmortem | `docs/reports/modal_deployment_postmortem.md` |