update: FAL results report with final eval numbers + conclusions
Browse files- docs/reports/fal_demo_results.md +102 -64
docs/reports/fal_demo_results.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
-
# Future-as-Label Demo: Results Report
|
| 2 |
|
| 3 |
**Date:** 2026-05-11
|
| 4 |
-
**Status:**
|
| 5 |
-
**Runtime:** 79
|
| 6 |
|
| 7 |
---
|
| 8 |
|
|
@@ -26,32 +26,36 @@
|
|
| 26 |
|
| 27 |
---
|
| 28 |
|
| 29 |
-
## 2. Evaluation Results
|
| 30 |
|
| 31 |
-
### Parse Rate β The Headline
|
| 32 |
|
| 33 |
| Model | Parse Rate | Meaning |
|
| 34 |
|---|---|---|
|
| 35 |
| **Base** | **0/134 (0%)** | Base model never outputs "PROB: X.XX" β completely ignores format instruction |
|
| 36 |
| **Tuned** | **134/134 (100%)** | GRPO taught perfect format compliance from zero |
|
| 37 |
|
| 38 |
-
|
| 39 |
|
| 40 |
-
###
|
| 41 |
|
| 42 |
-
|
|
| 43 |
-
|---|---|---|
|
| 44 |
-
|
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
-
|
| 52 |
-
|---|---|
|
| 53 |
-
| Base | 0.0075 | Meaningless β all predictions are exactly 0.5, so trivially "calibrated" to that bin |
|
| 54 |
-
| Tuned | 0.1740 | Model spreads predictions across bins but not perfectly calibrated |
|
| 55 |
|
| 56 |
---
|
| 57 |
|
|
@@ -59,81 +63,115 @@ This is the single most important result: **GRPO took a model with 0% format com
|
|
| 59 |
|
| 60 |
### What the model learned
|
| 61 |
|
| 62 |
-
1. **Format (complete success):** The model went from never producing "PROB:" to always producing it.
|
| 63 |
|
| 64 |
-
2. **Probability
|
| 65 |
|
| 66 |
-
3. **
|
| 67 |
|
| 68 |
-
### Why
|
| 69 |
|
| 70 |
-
The
|
|
|
|
|
|
|
| 71 |
|
| 72 |
-
|
| 73 |
-
- Tuned model produces varied predictions (0.2, 0.7, 0.8, etc.) but isn't perfectly calibrated
|
| 74 |
-
- A model that says "0.8" when the true rate is 0.6 gets penalized more than one that says "0.5"
|
| 75 |
|
| 76 |
-
|
| 77 |
|
| 78 |
-
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|---|---|---|
|
| 82 |
-
| Format compliance | 100% (from high baseline) | **0% β 100%** (from zero) |
|
| 83 |
-
| Brier improvement | 0.247 β 0.179 (β27%) | 0.25 β 0.29 (+16%) β miscalibrated |
|
| 84 |
-
| ECE improvement | 0.218 β 0.104 (β52%) | N/A β 0.174 |
|
| 85 |
-
| G (rollouts) | 4 | 2 (minimum viable) |
|
| 86 |
-
| Model capacity | 32B | 0.5B |
|
| 87 |
|
| 88 |
-
|
| 89 |
|
| 90 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
|
| 92 |
-
##
|
| 93 |
|
| 94 |
-
|
|
|
|
|
|
|
| 95 |
|
| 96 |
-
|
| 97 |
-
2. **More steps** (1500-3000): The model is at 0.53 epochs, still learning. V4.2 peaked at ~1.5 epochs.
|
| 98 |
-
3. **Larger model** (1.5B-3B): The paper's 32B result shows calibration improves with scale
|
| 99 |
-
4. **Log score reward** (paper's actual method): Harsher penalty for overconfidence may push calibration faster
|
| 100 |
|
| 101 |
---
|
| 102 |
|
| 103 |
-
## 5.
|
| 104 |
|
| 105 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
-
-
|
| 108 |
-
- No reward hacking (reward oscillated naturally, never spiked to 1.0)
|
| 109 |
-
- Completion length decreased organically (48β32 tokens) as model learned to be concise
|
| 110 |
-
- Step time decreased (6.2sβ5.5s) as completions got shorter
|
| 111 |
|
| 112 |
-
##
|
| 113 |
|
| 114 |
-
|
| 115 |
-
- Calibration refinement showed slow improvement (0.08β0.70 reward) but didn't translate to better Brier on eval
|
| 116 |
-
- The frac_zero_std was 0 after step 100, meaning GRPO always had signal β but with only 2 rollouts, the signal is noisy
|
| 117 |
|
| 118 |
-
|
| 119 |
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
|
| 124 |
---
|
| 125 |
|
| 126 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 127 |
|
| 128 |
-
|
| 129 |
|
| 130 |
-
The
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
---
|
| 133 |
|
| 134 |
-
##
|
| 135 |
|
| 136 |
-
|
| 137 |
-
-
|
| 138 |
-
|
| 139 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Future-as-Label Demo: Final Results Report
|
| 2 |
|
| 3 |
**Date:** 2026-05-11
|
| 4 |
+
**Status:** Complete β
(training + evaluation)
|
| 5 |
+
**Runtime:** 79 min training + 5 min eval on Tesla T4, ~$0.85 total cost
|
| 6 |
|
| 7 |
---
|
| 8 |
|
|
|
|
| 26 |
|
| 27 |
---
|
| 28 |
|
| 29 |
+
## 2. Final Evaluation Results
|
| 30 |
|
| 31 |
+
### Parse Rate β The Headline Result
|
| 32 |
|
| 33 |
| Model | Parse Rate | Meaning |
|
| 34 |
|---|---|---|
|
| 35 |
| **Base** | **0/134 (0%)** | Base model never outputs "PROB: X.XX" β completely ignores format instruction |
|
| 36 |
| **Tuned** | **134/134 (100%)** | GRPO taught perfect format compliance from zero |
|
| 37 |
|
| 38 |
+
**GRPO took a model with 0% format compliance and achieved 100% in 800 steps (79 minutes, $0.85).**
|
| 39 |
|
| 40 |
+
### Overall Metrics
|
| 41 |
|
| 42 |
+
| Metric | Base | Tuned | Ξ | Interpretation |
|
| 43 |
+
|---|---|---|---|---|
|
| 44 |
+
| **Brier β** | 0.2500 | 0.2653 | +0.0153 | Tuned slightly worse (but base is artificial 0.5 fallback) |
|
| 45 |
+
| **ECE β** | 0.0075 | 0.1394 | +0.1319 | Tuned spreads predictions across range; not perfectly calibrated |
|
| 46 |
+
| **Base rate Brier** | 0.2499 | β | β | Always-predict-base-rate baseline |
|
| 47 |
+
| **Base rate** | 0.4925 | β | β | ~49% positive rate (balanced) |
|
| 48 |
+
| **Wilcoxon p** | β | 0.9495 | β | β Not significant |
|
| 49 |
|
| 50 |
+
### Per-Task Breakdown
|
| 51 |
|
| 52 |
+
| Task | Base Brier | Tuned Brier | Ξ | N |
|
| 53 |
+
|---|---|---|---|---|
|
| 54 |
+
| repurchase_90d | N/A | N/A | β | 0 (no eval samples generated for this task) |
|
| 55 |
+
| review_positive | 0.2500 | 0.2723 | +0.0223 | 67 |
|
| 56 |
+
| delivery_ontime | 0.2500 | 0.2583 | +0.0083 | 67 |
|
| 57 |
|
| 58 |
+
**Note:** repurchase_90d has 0 eval samples β a bug in the data pipeline's temporal censoring filter (the 90-day window extends past the eval end date, filtering out all eval candidates). This means only 2 of 3 tasks were evaluated.
|
|
|
|
|
|
|
|
|
|
| 59 |
|
| 60 |
---
|
| 61 |
|
|
|
|
| 63 |
|
| 64 |
### What the model learned
|
| 65 |
|
| 66 |
+
1. **Format (complete success):** 0% β 100% parse rate. The model went from never producing "PROB:" to always producing it. Confirms V4.2's finding: GRPO teaches format through reward alone.
|
| 67 |
|
| 68 |
+
2. **Probability discrimination (partial):** The model produces varied probabilities (not a constant), as evidenced by reward_std > 0 and ECE > 0. It IS using the input features to modulate its predictions.
|
| 69 |
|
| 70 |
+
3. **Calibration (not achieved):** Tuned Brier (0.265) is slightly worse than the base fallback (0.250). The model's probability estimates don't yet align with true event frequencies.
|
| 71 |
|
| 72 |
+
### Why the base "wins" on Brier
|
| 73 |
|
| 74 |
+
The base Brier of 0.25 is **not a real prediction** β it's the mathematical result of:
|
| 75 |
+
- 0% parse rate β all predictions fall back to 0.5
|
| 76 |
+
- On balanced data (base_rate=0.49): (0.5 - y)Β² averages to 0.25
|
| 77 |
|
| 78 |
+
This is the **optimal constant predictor** for balanced binary data. Any model that actually tries to differentiate (predict 0.8 for some, 0.2 for others) will score worse unless its predictions are well-calibrated. The tuned model at Brier=0.265 is only +0.015 above this floor β meaning it's approximately correct on average but slightly overconfident on incorrect predictions.
|
|
|
|
|
|
|
| 79 |
|
| 80 |
+
### The delivery_ontime task nearly matches baseline
|
| 81 |
|
| 82 |
+
Delivery task Brier: 0.2583 (only +0.008 above 0.25). This is very close to the optimal constant β the model may have learned to predict near-0.5 for this task (appropriate for balanced data) while spreading predictions for review_positive (0.2723, slightly worse).
|
| 83 |
|
| 84 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
+
## 4. Comparison to Paper & V4.2
|
| 87 |
|
| 88 |
+
| Dimension | Paper (32B) | V4.2 Commerce | FAL Demo (0.5B) |
|
| 89 |
+
|---|---|---|---|
|
| 90 |
+
| Format learning | N/A (already formatted) | +29.5% extraction | **0%β100% parse** |
|
| 91 |
+
| Calibration | β27% Brier | N/A | +6% Brier (slightly worse) |
|
| 92 |
+
| Model | Qwen3-32B | Tucano2-0.5B | Tucano2-0.5B |
|
| 93 |
+
| G (rollouts) | 4 | 16 | 2 |
|
| 94 |
+
| Training steps | 160 | 1,100 (best) | 800 |
|
| 95 |
+
| Cost | ~$100+ (est) | ~$18 | **$0.85** |
|
| 96 |
|
| 97 |
+
### Key insight: format learning is the consistent GRPO win at 0.5B
|
| 98 |
|
| 99 |
+
Both V4.2 and FAL demo show the same pattern:
|
| 100 |
+
- **Format/structure learning:** massive improvement (JSON types, probability format)
|
| 101 |
+
- **Content/calibration quality:** marginal or no improvement at 0.5B
|
| 102 |
|
| 103 |
+
GRPO at this scale is fundamentally a **formatting tool**, not a reasoning tool.
|
|
|
|
|
|
|
|
|
|
| 104 |
|
| 105 |
---
|
| 106 |
|
| 107 |
+
## 5. What Would Improve Calibration
|
| 108 |
|
| 109 |
+
| Change | Expected impact | Evidence |
|
| 110 |
+
|---|---|---|
|
| 111 |
+
| G=8+ (more rollouts) | Strong | Paper uses G=4; our G=2 gives 70% zero-std early on |
|
| 112 |
+
| 1.5B-3B model | Strong | Paper used 32B; 0.5B may lack capacity for probability reasoning |
|
| 113 |
+
| Log score reward | Moderate | Harsher penalty for overconfidence (paper's actual method) |
|
| 114 |
+
| More training (1500+ steps) | Moderate | Only 0.53 epochs completed; V4.2 peaked at 1.5 |
|
| 115 |
+
| Fix repurchase task in eval | Minor | Adds 67 more eval samples for significance |
|
| 116 |
+
| L40S GPU + G=16 batch=16 | Strong | Matches V4.2 signal strength |
|
| 117 |
|
| 118 |
+
---
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
+
## 6. Technical Findings
|
| 121 |
|
| 122 |
+
### Data pipeline bug: repurchase_90d has 0 eval samples
|
|
|
|
|
|
|
| 123 |
|
| 124 |
+
The temporal censoring filter (`order_purchase_timestamp <= dataset_end - 90 days`) combined with the eval window (2018-06-01 to 2018-08-31) leaves zero observable repurchase outcomes β because you need 90 days AFTER the order to observe whether repurchase happened, but the dataset ends before that window closes for eval-period orders.
|
| 125 |
|
| 126 |
+
**Fix for next run:** Either extend the eval end date or shorten the repurchase window to 30 days for the eval split.
|
| 127 |
+
|
| 128 |
+
### Training was remarkably stable
|
| 129 |
+
|
| 130 |
+
- No entropy collapse (entropy 1.5-2.0 throughout all 800 steps)
|
| 131 |
+
- No reward hacking
|
| 132 |
+
- Completion length decreased organically (48β32 tokens)
|
| 133 |
+
- No gradient explosion (grad_norm 0-2.5, healthy range)
|
| 134 |
+
|
| 135 |
+
### The "trivially calibrated" base model problem
|
| 136 |
+
|
| 137 |
+
When base parse rate = 0%, evaluating calibration is meaningless β the base model isn't making predictions, it's just outputting text. The only fair comparison would be against a **prompted** base model that CAN produce probabilities (e.g., with few-shot examples). This wasn't implemented.
|
| 138 |
|
| 139 |
---
|
| 140 |
|
| 141 |
+
## 7. Conclusions
|
| 142 |
+
|
| 143 |
+
### β
Success: Format learning generalizes
|
| 144 |
+
|
| 145 |
+
GRPO + reward shaping teaches format compliance on any output structure:
|
| 146 |
+
- V4.2: JSON with correct types (+29.5%)
|
| 147 |
+
- FAL: Probability decimal format (0% β 100%)
|
| 148 |
+
- Both: ~80-100 steps to learn, ~$1 cost on T4
|
| 149 |
|
| 150 |
+
### β οΈ Partial: Probability production without calibration
|
| 151 |
|
| 152 |
+
The model produces varied probabilities (not a constant) β it IS differentiating between inputs. But the predictions aren't calibrated enough to beat the naive 0.5 baseline. This is the expected outcome for G=2 on 0.5B: **necessary but not sufficient signal for calibration learning**.
|
| 153 |
+
|
| 154 |
+
### β Not achieved: Brier improvement over baseline
|
| 155 |
+
|
| 156 |
+
The success criterion "tuned Brier < base Brier" was not met. The Wilcoxon test confirms: p=0.95, no significant difference. This is the "0.5B below minimum viable scale for calibration" outcome from ADR-003.
|
| 157 |
+
|
| 158 |
+
### The experiment is still informative
|
| 159 |
+
|
| 160 |
+
We now know:
|
| 161 |
+
1. **Lower bound for calibration:** 0.5B with G=2 is below the threshold
|
| 162 |
+
2. **Format learning works universally:** Another domain, same result as V4.2
|
| 163 |
+
3. **Stability is not an issue:** 800 steps without collapse (vs. V4.2's collapse at step 1100)
|
| 164 |
+
4. **Cost:** $0.85 for a complete GRPO experiment including eval β this workflow is cheap enough for daily iteration
|
| 165 |
|
| 166 |
---
|
| 167 |
|
| 168 |
+
## 8. Files
|
| 169 |
|
| 170 |
+
| File | Contents |
|
| 171 |
+
|---|---|
|
| 172 |
+
| Model checkpoint | Modal volume `fal-checkpoints` β `/checkpoints/fal-grpo/final/` |
|
| 173 |
+
| Eval data | Modal volume β `/checkpoints/fal-grpo/eval_dataset.json` |
|
| 174 |
+
| Results JSON | Modal volume β `/checkpoints/fal-grpo/fal_results.json` |
|
| 175 |
+
| Training script | `fal_demo/train_modal.py` |
|
| 176 |
+
| ADR | `docs/ADR-003-future-as-label.md` |
|
| 177 |
+
| Deployment postmortem | `docs/reports/modal_deployment_postmortem.md` |
|