add: FAL demo results report (preliminary — eval crash pending fix)
Browse files- docs/reports/fal_demo_results.md +139 -0
docs/reports/fal_demo_results.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Future-as-Label Demo: Results Report
|
| 2 |
+
|
| 3 |
+
**Date:** 2026-05-11
|
| 4 |
+
**Status:** Training complete ✅, Evaluation partially complete (crash in per-task ECE computation)
|
| 5 |
+
**Runtime:** 79 minutes on Tesla T4, ~$0.80 total cost
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## 1. Training Summary
|
| 10 |
+
|
| 11 |
+
| Phase | Steps | Reward | What happened |
|
| 12 |
+
|---|---|---|---|
|
| 13 |
+
| 0–11 | Warmup | 0.08 | 8% parse rate, model can't produce "PROB:" format |
|
| 14 |
+
| 11–65 | Format learning | 0.08→0.30 | Breakthrough — model discovers parseable format |
|
| 15 |
+
| 65–100 | Format consolidation | 0.30→0.70 | 100% parse rate achieved, frac_zero_std drops to 0 |
|
| 16 |
+
| 100–800 | Calibration refinement | 0.65–0.83 | Stable plateau, model refining probability estimates |
|
| 17 |
+
|
| 18 |
+
**Final training metrics (step 800):**
|
| 19 |
+
- reward: 0.62–0.75 (oscillating, healthy)
|
| 20 |
+
- frac_reward_zero_std: 0 (every batch has signal)
|
| 21 |
+
- entropy: 1.6–2.0 (stable, no collapse)
|
| 22 |
+
- clipped_ratio: 0.5–0.8 (model learned shorter outputs)
|
| 23 |
+
- completions/mean_length: decreasing from 48→32 tokens (learning efficiency)
|
| 24 |
+
- epoch: 0.53 (seen ~half the data once)
|
| 25 |
+
- train_loss: 0.0011
|
| 26 |
+
|
| 27 |
+
---
|
| 28 |
+
|
| 29 |
+
## 2. Evaluation Results (Partial)
|
| 30 |
+
|
| 31 |
+
### Parse Rate — The Headline Finding
|
| 32 |
+
|
| 33 |
+
| Model | Parse Rate | Meaning |
|
| 34 |
+
|---|---|---|
|
| 35 |
+
| **Base** | **0/134 (0%)** | Base model never outputs "PROB: X.XX" — completely ignores format instruction |
|
| 36 |
+
| **Tuned** | **134/134 (100%)** | GRPO taught perfect format compliance from zero |
|
| 37 |
+
|
| 38 |
+
This is the single most important result: **GRPO took a model with 0% format compliance and achieved 100% in 800 steps (79 minutes, $0.80).** This replicates the V4.2 finding (extraction JSON format learning) in a completely different domain.
|
| 39 |
+
|
| 40 |
+
### Brier Score (Overall)
|
| 41 |
+
|
| 42 |
+
| Model | Brier ↓ | Interpretation |
|
| 43 |
+
|---|---|---|
|
| 44 |
+
| Base | 0.2500 | Always predicts 0.5 (fallback for unparseable) → (0.5−y)² on balanced data = 0.25 |
|
| 45 |
+
| Tuned | 0.2905 | Produces real probabilities but slightly miscalibrated |
|
| 46 |
+
|
| 47 |
+
**Important context:** The base "0.25" is not a real prediction — it's the artifact of 0% parse rate falling back to 0.5 on a balanced (50/50) dataset. The fair comparison is tuned model vs. **base rate predictor** (always predict the marginal probability per task).
|
| 48 |
+
|
| 49 |
+
### ECE (Overall)
|
| 50 |
+
|
| 51 |
+
| Model | ECE ↓ |
|
| 52 |
+
|---|---|
|
| 53 |
+
| Base | 0.0075 | Meaningless — all predictions are exactly 0.5, so trivially "calibrated" to that bin |
|
| 54 |
+
| Tuned | 0.1740 | Model spreads predictions across bins but not perfectly calibrated |
|
| 55 |
+
|
| 56 |
+
---
|
| 57 |
+
|
| 58 |
+
## 3. Interpretation
|
| 59 |
+
|
| 60 |
+
### What the model learned
|
| 61 |
+
|
| 62 |
+
1. **Format (complete success):** The model went from never producing "PROB:" to always producing it. This confirms V4.2's finding: GRPO can teach output format through reward shaping alone.
|
| 63 |
+
|
| 64 |
+
2. **Probability values (partial):** The model produces different probabilities for different inputs (reward_std > 0 throughout), meaning it's not just outputting a constant. But the Brier score (0.29) being worse than naive 0.5 means its probabilities are miscalibrated — likely overconfident on wrong predictions.
|
| 65 |
+
|
| 66 |
+
3. **No entropy collapse:** Unlike V4.2 (which collapsed at 1.5 epochs), this run completed 0.53 epochs without collapse. The simpler output format (just a number) and shorter completions prevent the diversity loss that killed V4.2.
|
| 67 |
+
|
| 68 |
+
### Why calibration is worse than naive baseline
|
| 69 |
+
|
| 70 |
+
The tuned Brier (0.29) > base Brier (0.25) doesn't mean the model is "worse" — it means:
|
| 71 |
+
|
| 72 |
+
- Base always predicts 0.5 → on balanced data, this is optimal in expectation (minimizes expected Brier)
|
| 73 |
+
- Tuned model produces varied predictions (0.2, 0.7, 0.8, etc.) but isn't perfectly calibrated
|
| 74 |
+
- A model that says "0.8" when the true rate is 0.6 gets penalized more than one that says "0.5"
|
| 75 |
+
|
| 76 |
+
This is the **overconfidence problem** the Future-as-Label paper addresses with more training (160 steps on 32B vs our 800 steps on 0.5B). With G=2 (minimal rollouts), the calibration signal is weak — the model learned format but doesn't have enough reward variance to learn precise calibration.
|
| 77 |
+
|
| 78 |
+
### Comparison to paper results
|
| 79 |
+
|
| 80 |
+
| | Paper (32B, 160 steps) | Our demo (0.5B, 800 steps) |
|
| 81 |
+
|---|---|---|
|
| 82 |
+
| Format compliance | 100% (from high baseline) | **0% → 100%** (from zero) |
|
| 83 |
+
| Brier improvement | 0.247 → 0.179 (−27%) | 0.25 → 0.29 (+16%) — miscalibrated |
|
| 84 |
+
| ECE improvement | 0.218 → 0.104 (−52%) | N/A → 0.174 |
|
| 85 |
+
| G (rollouts) | 4 | 2 (minimum viable) |
|
| 86 |
+
| Model capacity | 32B | 0.5B |
|
| 87 |
+
|
| 88 |
+
**Expected:** A 0.5B model with G=2 was always going to be limited on calibration — the research question was "does it learn anything?" and the answer is definitively yes (format learning), partially (probability production), but not yet (calibration).
|
| 89 |
+
|
| 90 |
+
---
|
| 91 |
+
|
| 92 |
+
## 4. What Would Improve Results
|
| 93 |
+
|
| 94 |
+
Based on V4.2 lessons + paper methodology:
|
| 95 |
+
|
| 96 |
+
1. **G=8 or G=16** (requires larger GPU or batch adjustment): More rollouts per prompt = more reward variance = stronger calibration signal
|
| 97 |
+
2. **More steps** (1500-3000): The model is at 0.53 epochs, still learning. V4.2 peaked at ~1.5 epochs.
|
| 98 |
+
3. **Larger model** (1.5B-3B): The paper's 32B result shows calibration improves with scale
|
| 99 |
+
4. **Log score reward** (paper's actual method): Harsher penalty for overconfidence may push calibration faster
|
| 100 |
+
|
| 101 |
+
---
|
| 102 |
+
|
| 103 |
+
## 5. Technical Findings
|
| 104 |
+
|
| 105 |
+
### Training was remarkably stable
|
| 106 |
+
|
| 107 |
+
- No entropy collapse (entropy stayed 1.5-2.0 throughout)
|
| 108 |
+
- No reward hacking (reward oscillated naturally, never spiked to 1.0)
|
| 109 |
+
- Completion length decreased organically (48→32 tokens) as model learned to be concise
|
| 110 |
+
- Step time decreased (6.2s→5.5s) as completions got shorter
|
| 111 |
+
|
| 112 |
+
### G=2 provides enough signal for format learning, not for calibration
|
| 113 |
+
|
| 114 |
+
- Format learning happened in ~80 steps (0→100% parse rate)
|
| 115 |
+
- Calibration refinement showed slow improvement (0.08→0.70 reward) but didn't translate to better Brier on eval
|
| 116 |
+
- The frac_zero_std was 0 after step 100, meaning GRPO always had signal — but with only 2 rollouts, the signal is noisy
|
| 117 |
+
|
| 118 |
+
### T4 is sufficient for 0.5B GRPO without mixed precision
|
| 119 |
+
|
| 120 |
+
- Peak VRAM: ~4GB (model) + ~2GB (generations) ≈ 6GB of 16GB used
|
| 121 |
+
- No OOM, no bf16 issues after disabling mixed precision
|
| 122 |
+
- Cost: $0.80 for full 800-step run — viable for rapid iteration
|
| 123 |
+
|
| 124 |
+
---
|
| 125 |
+
|
| 126 |
+
## 6. Conclusion
|
| 127 |
+
|
| 128 |
+
**The Future-as-Label PoC succeeded on format learning and failed on calibration.** This is the "format learned but no calibration" outcome predicted in ADR-003 — it demonstrates that GRPO can teach a 0.5B model to output probabilities (from 0% → 100% format compliance), but the model doesn't have sufficient capacity + signal to learn *well-calibrated* probabilities with only G=2 rollouts.
|
| 129 |
+
|
| 130 |
+
The result is scientifically informative: it establishes a lower bound on what's needed for calibration learning, and confirms that format-learning generalizes across tasks (JSON in V4.2, probabilities here).
|
| 131 |
+
|
| 132 |
+
---
|
| 133 |
+
|
| 134 |
+
## 7. Pending
|
| 135 |
+
|
| 136 |
+
- [ ] Fix eval crash (ZeroDivisionError in per-task ECE when a task has no base predictions in a bin)
|
| 137 |
+
- [ ] Re-run eval only (no retraining needed — model saved to volume)
|
| 138 |
+
- [ ] Add per-task breakdown to determine which tasks the model calibrates best on
|
| 139 |
+
- [ ] Compare tuned Brier to base-rate Brier (always predict marginal) — the true baseline
|