CreativeEngineer commited on
Commit
80ce5e5
·
1 Parent(s): 92c59c2

docs: polish verifier reward review

Browse files
docs/VERIFIER_REWARD_REVIEW_2026-03-08.md CHANGED
@@ -13,10 +13,10 @@ This note reviews how well the current verifier path and reward function serve t
13
 
14
  Primary review targets:
15
 
16
- - [docs/FUSION_DESIGN_LAB_PLAN_V2.md](/Users/suhjungdae/code/fusion-design-lab/docs/FUSION_DESIGN_LAB_PLAN_V2.md)
17
- - [docs/P1_ENV_CONTRACT_V1.md](/Users/suhjungdae/code/fusion-design-lab/docs/P1_ENV_CONTRACT_V1.md)
18
- - [server/physics.py](/Users/suhjungdae/code/fusion-design-lab/server/physics.py)
19
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py)
20
  - recent history from `d58c100` through `6deaccc`
21
 
22
  ## Executive Summary
@@ -48,10 +48,9 @@ The planning docs explicitly say the environment artifact is the product, not ju
48
 
49
  Relevant references:
50
 
51
- - [docs/FUSION_DESIGN_LAB_PLAN_V2.md](/Users/suhjungdae/code/fusion-design-lab/docs/FUSION_DESIGN_LAB_PLAN_V2.md#L36)
52
- - [docs/FUSION_DESIGN_LAB_PLAN_V2.md](/Users/suhjungdae/code/fusion-design-lab/docs/FUSION_DESIGN_LAB_PLAN_V2.md#L420)
53
- - [docs/FUSION_DELIVERABLES_MAP.md](/Users/suhjungdae/code/fusion-design-lab/docs/FUSION_DELIVERABLES_MAP.md#L21)
54
- - [server/data/p1/README.md](/Users/suhjungdae/code/fusion-design-lab/server/data/p1/README.md#L15)
55
 
56
  ### 2. Observation legibility still conflicts with the official tolerance semantics
57
 
@@ -67,21 +66,23 @@ This is a direct usability problem for a repo whose goal is a legible, human-pla
67
 
68
  Relevant references:
69
 
70
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py#L291)
71
- - [server/physics.py](/Users/suhjungdae/code/fusion-design-lab/server/physics.py#L114)
72
- - [docs/P1_PARAMETERIZATION_DEEPDIVE.md](/Users/suhjungdae/code/fusion-design-lab/docs/P1_PARAMETERIZATION_DEEPDIVE.md#L202)
73
 
74
  ### 3. High-fidelity best-state bookkeeping is still not trustworthy
75
 
76
  `best_high_fidelity_score` and `best_high_fidelity_feasibility` are not true best high-fidelity values. On submit, the env can overwrite them by reevaluating `best_params`, even if the current submit is the best actual high-fidelity result.
77
 
 
 
78
  This weakens the clean low-fi/high-fi split the contract calls for.
79
 
80
  Relevant references:
81
 
82
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py#L368)
83
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py#L449)
84
- - [docs/P1_ENV_CONTRACT_V1.md](/Users/suhjungdae/code/fusion-design-lab/docs/P1_ENV_CONTRACT_V1.md#L163)
85
 
86
  ### 4. Submit is operationally expensive
87
 
@@ -91,35 +92,48 @@ One `submit` can trigger up to three high-fidelity evaluations:
91
  2. initial high-fidelity reference
92
  3. reevaluated `best_params`
93
 
 
 
94
  That makes baseline comparison and manual validation much slower than the task surface implies.
95
 
96
  Relevant references:
97
 
98
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py#L151)
99
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py#L442)
 
 
100
 
101
- ### 5. Baseline comparison does not reject failed high-fidelity submits
102
 
103
- [baselines/compare.py](/Users/suhjungdae/code/fusion-design-lab/baselines/compare.py#L54) only checks that the final step used high-fidelity evaluation. It does not check whether that high-fidelity submit succeeded.
 
 
 
 
 
 
 
 
 
104
 
105
  That means a failed submit can still count as a terminal high-fi result in comparison output.
106
 
107
- ### 6. Verifier failure normalization is incomplete
108
 
109
- [server/physics.py](/Users/suhjungdae/code/fusion-design-lab/server/physics.py#L87) converts `RuntimeError` into explicit failure metrics, but other exception types can still crash the env instead of producing documented failure observations.
110
 
111
- ### 7. Submit budget reporting is inconsistent
112
 
113
  `submit` does not decrement `budget_remaining`, so the terminal observation shows pre-submit budget. Since `submit` is terminal, this is not a serious episode-flow bug, but it is a contract/reporting inconsistency.
114
 
115
  Relevant reference:
116
 
117
- - [server/environment.py](/Users/suhjungdae/code/fusion-design-lab/server/environment.py#L151)
118
 
119
- ### 8. Minor maintenance smells
120
 
121
- - [server/app.py](/Users/suhjungdae/code/fusion-design-lab/server/app.py#L6) imports `N_FIELD_PERIODS` indirectly via `server.environment` instead of directly from `server.contract`
122
- - [fusion_lab/models.py](/Users/suhjungdae/code/fusion-design-lab/fusion_lab/models.py#L57) and [fusion_lab/models.py](/Users/suhjungdae/code/fusion-design-lab/fusion_lab/models.py#L62) have misleading defaults if used outside the environment
123
 
124
  ## Findings From The Quoted External Review
125
 
@@ -135,7 +149,7 @@ Relevant reference:
135
 
136
  - submit budget reporting is inconsistent, but it does not allow the agent to exceed the episode budget because submit is terminal
137
 
138
- ### Not Confirmed
139
 
140
  The quoted review's highest-severity claim was an `_update_best` feasibility mismatch between:
141
 
@@ -144,7 +158,16 @@ The quoted review's highest-severity claim was an `_update_best` feasibility mis
144
 
145
  I did not reproduce that as a current live bug. I sampled 30 low-fidelity evaluations across the current parameter ranges and found zero mismatches between those two criteria.
146
 
147
- That issue is only real if upstream breaks the invariant between `GeometricalProblem.is_feasible()` and `compute_feasibility()`. On current `HEAD`, it should not be treated as the top issue.
 
 
 
 
 
 
 
 
 
148
 
149
  ## Validation Performed
150
 
@@ -166,16 +189,33 @@ So the task is not trivially solved at reset.
166
 
167
  ### Sampled low-fidelity trajectory
168
 
169
- For seed `0`, a simple heuristic-like low-fidelity sequence reached feasibility within 5 steps, and the feasibility-crossing step received a strong positive reward (`+3.1054`).
 
 
 
 
 
 
 
 
170
 
171
  This is a good sign for current `Reward V0` ordering, but it is not a substitute for the fixture/manual-playtest loop the docs require.
172
 
 
 
 
 
 
 
 
 
 
173
  ### Current workspace state at review time
174
 
175
  The workspace was dirty during review:
176
 
177
- - modified [training/notebooks/northflank_smoke.py](/Users/suhjungdae/code/fusion-design-lab/training/notebooks/northflank_smoke.py)
178
- - untracked [baselines/measured_sweep.py](/Users/suhjungdae/code/fusion-design-lab/baselines/measured_sweep.py)
179
  - untracked `baselines/sweep_results/`
180
 
181
  These were treated as workspace observations, not committed `HEAD` findings.
@@ -235,6 +275,7 @@ Weaknesses:
235
 
236
  - not yet validated by the playtest/fixture loop the repo requires
237
  - some reward/reporting details still depend on unclear high-fi bookkeeping
 
238
 
239
  ## TODOs
240
 
@@ -242,12 +283,12 @@ Weaknesses:
242
 
243
  1. Fix diagnostics so humans can understand official tolerance-based feasibility.
244
  2. Fix true high-fidelity best-state bookkeeping.
245
- 3. Update [baselines/compare.py](/Users/suhjungdae/code/fusion-design-lab/baselines/compare.py) to reject or explicitly flag failed high-fidelity submits.
246
  4. Reduce submit-time high-fidelity reevaluation cost.
247
 
248
  ### Validation Priority
249
 
250
- 5. Add tracked `P1` fixtures under [server/data/p1/](/Users/suhjungdae/code/fusion-design-lab/server/data/p1/).
251
  6. Run fixture sanity checks for:
252
  - good or near-boundary cases
253
  - clearly bad cases
@@ -261,12 +302,14 @@ Weaknesses:
261
  8. Either:
262
  - document the first real `Reward V0 -> V1` pathology/fix
263
  - or explicitly record that `Reward V0` survived playtesting unchanged
 
 
264
 
265
  ### Secondary Cleanup
266
 
267
- 9. Decide whether submit should decrement `budget_remaining` for contract clarity.
268
- 10. Import `N_FIELD_PERIODS` in [server/app.py](/Users/suhjungdae/code/fusion-design-lab/server/app.py) directly from `server.contract`.
269
- 11. Clean up or intentionally commit the current workspace-only baseline and notebook artifacts.
270
 
271
  ## Bottom Line
272
 
 
13
 
14
  Primary review targets:
15
 
16
+ - [FUSION_DESIGN_LAB_PLAN_V2.md](./FUSION_DESIGN_LAB_PLAN_V2.md)
17
+ - [P1_ENV_CONTRACT_V1.md](./P1_ENV_CONTRACT_V1.md)
18
+ - [server/physics.py](../server/physics.py)
19
+ - [server/environment.py](../server/environment.py)
20
  - recent history from `d58c100` through `6deaccc`
21
 
22
  ## Executive Summary
 
48
 
49
  Relevant references:
50
 
51
+ - [FUSION_DESIGN_LAB_PLAN_V2.md](./FUSION_DESIGN_LAB_PLAN_V2.md)
52
+ - [FUSION_DELIVERABLES_MAP.md](./FUSION_DELIVERABLES_MAP.md)
53
+ - [server/data/p1/README.md](../server/data/p1/README.md)
 
54
 
55
  ### 2. Observation legibility still conflicts with the official tolerance semantics
56
 
 
66
 
67
  Relevant references:
68
 
69
+ - [server/environment.py](../server/environment.py) in `_build_observation(...)`
70
+ - [server/physics.py](../server/physics.py) in `_to_evaluation_metrics(...)`
71
+ - [P1_PARAMETERIZATION_DEEPDIVE.md](./P1_PARAMETERIZATION_DEEPDIVE.md)
72
 
73
  ### 3. High-fidelity best-state bookkeeping is still not trustworthy
74
 
75
  `best_high_fidelity_score` and `best_high_fidelity_feasibility` are not true best high-fidelity values. On submit, the env can overwrite them by reevaluating `best_params`, even if the current submit is the best actual high-fidelity result.
76
 
77
+ The worst failure mode is sharper than simple bookkeeping drift: if reevaluating `best_params` crashes VMEC, the env can store failure-sentinel data under `best_high_fidelity_*`, including `p1_score=0.0`, while still labeling that state as "best."
78
+
79
  This weakens the clean low-fi/high-fi split the contract calls for.
80
 
81
  Relevant references:
82
 
83
+ - [server/environment.py](../server/environment.py) in `_refresh_best_high_fidelity_metrics(...)`
84
+ - [server/environment.py](../server/environment.py) in `_summary_submit(...)`
85
+ - [P1_ENV_CONTRACT_V1.md](./P1_ENV_CONTRACT_V1.md)
86
 
87
  ### 4. Submit is operationally expensive
88
 
 
92
  2. initial high-fidelity reference
93
  3. reevaluated `best_params`
94
 
95
+ The initial high-fidelity reference cache is only per episode because it lives on `State`, which is recreated on every `reset()`. In practice, repeated episodes still repay that cost.
96
+
97
  That makes baseline comparison and manual validation much slower than the task surface implies.
98
 
99
  Relevant references:
100
 
101
+ - [server/environment.py](../server/environment.py) in `_handle_submit(...)`
102
+ - [server/environment.py](../server/environment.py) in `_initial_high_fidelity_score(...)`
103
+
104
+ ### 5. Heuristic baseline evidence is stale and currently unconvincing
105
 
106
+ The current heuristic is not validated on the repaired 4-knob family and is already documented in repo status notes as underperforming random on the real verifier path.
107
 
108
+ This is not a verifier bug, but it is a serious artifact-quality problem for a repo that needs credible baseline evidence.
109
+
110
+ Relevant references:
111
+
112
+ - [README.md](../README.md)
113
+ - [TODO.md](../TODO.md)
114
+
115
+ ### 6. Baseline comparison does not reject failed high-fidelity submits
116
+
117
+ [baselines/compare.py](../baselines/compare.py) in `_require_submit_fidelity(...)` only checks that the final step used high-fidelity evaluation. It does not check whether that high-fidelity submit succeeded.
118
 
119
  That means a failed submit can still count as a terminal high-fi result in comparison output.
120
 
121
+ ### 7. Verifier failure normalization is incomplete
122
 
123
+ [server/physics.py](../server/physics.py) in `evaluate_boundary(...)` converts `RuntimeError` into explicit failure metrics, but other exception types can still crash the env instead of producing documented failure observations.
124
 
125
+ ### 8. Submit budget reporting is inconsistent
126
 
127
  `submit` does not decrement `budget_remaining`, so the terminal observation shows pre-submit budget. Since `submit` is terminal, this is not a serious episode-flow bug, but it is a contract/reporting inconsistency.
128
 
129
  Relevant reference:
130
 
131
+ - [server/environment.py](../server/environment.py) in `_handle_submit(...)`
132
 
133
+ ### 9. Minor maintenance smells
134
 
135
+ - [server/app.py](../server/app.py) imports `N_FIELD_PERIODS` indirectly via `server.environment` instead of directly from `server.contract`
136
+ - [fusion_lab/models.py](../fusion_lab/models.py) has misleading defaults if used outside the environment
137
 
138
  ## Findings From The Quoted External Review
139
 
 
149
 
150
  - submit budget reporting is inconsistent, but it does not allow the agent to exceed the episode budget because submit is terminal
151
 
152
+ ### Fragile Coupling, Not A Reproduced Live Bug
153
 
154
  The quoted review's highest-severity claim was an `_update_best` feasibility mismatch between:
155
 
 
158
 
159
  I did not reproduce that as a current live bug. I sampled 30 low-fidelity evaluations across the current parameter ranges and found zero mismatches between those two criteria.
160
 
161
+ But the current code still represents a fragile coupling: `_update_best(...)` uses one concept through `constraints_satisfied` for `current`, and a separate hardcoded `FEASIBILITY_TOLERANCE` comparison for `best`. Those happen to agree today because current `constellaration` semantics line up with `compute_feasibility() <= 0.01`.
162
+
163
+ That means this is best described as:
164
+
165
+ - not a reproduced live bug on current `HEAD`
166
+ - still a maintainability risk if upstream tolerance semantics ever drift
167
+
168
+ Relevant reference:
169
+
170
+ - [server/environment.py](../server/environment.py) in `_update_best(...)`
171
 
172
  ## Validation Performed
173
 
 
189
 
190
  ### Sampled low-fidelity trajectory
191
 
192
+ For seed `0`, the following low-fidelity sequence reached feasibility within 5 steps:
193
+
194
+ 1. `triangularity_scale increase small`
195
+ 2. `triangularity_scale increase small`
196
+ 3. `triangularity_scale increase small`
197
+ 4. `rotational_transform increase small`
198
+ 5. `rotational_transform increase small`
199
+
200
+ The feasibility-crossing step received a strong positive reward (`+3.1054`).
201
 
202
  This is a good sign for current `Reward V0` ordering, but it is not a substitute for the fixture/manual-playtest loop the docs require.
203
 
204
+ One additional playtest note: reward stacking should be checked explicitly. A single step can combine:
205
+
206
+ - recovery bonus `+1.0`
207
+ - feasibility transition bonus `+3.0`
208
+ - shaping delta
209
+ - non-submit step cost `-0.1`
210
+
211
+ That is not obviously wrong, but it can produce a `+4.0+` step and should be verified during manual playtesting.
212
+
213
  ### Current workspace state at review time
214
 
215
  The workspace was dirty during review:
216
 
217
+ - modified [training/notebooks/northflank_smoke.py](../training/notebooks/northflank_smoke.py)
218
+ - untracked [baselines/measured_sweep.py](../baselines/measured_sweep.py)
219
  - untracked `baselines/sweep_results/`
220
 
221
  These were treated as workspace observations, not committed `HEAD` findings.
 
275
 
276
  - not yet validated by the playtest/fixture loop the repo requires
277
  - some reward/reporting details still depend on unclear high-fi bookkeeping
278
+ - reward stacking is still under-documented and should be checked during playtesting
279
 
280
  ## TODOs
281
 
 
283
 
284
  1. Fix diagnostics so humans can understand official tolerance-based feasibility.
285
  2. Fix true high-fidelity best-state bookkeeping.
286
+ 3. Update [baselines/compare.py](../baselines/compare.py) to reject or explicitly flag failed high-fidelity submits.
287
  4. Reduce submit-time high-fidelity reevaluation cost.
288
 
289
  ### Validation Priority
290
 
291
+ 5. Add tracked `P1` fixtures under [server/data/p1/](../server/data/p1/).
292
  6. Run fixture sanity checks for:
293
  - good or near-boundary cases
294
  - clearly bad cases
 
302
  8. Either:
303
  - document the first real `Reward V0 -> V1` pathology/fix
304
  - or explicitly record that `Reward V0` survived playtesting unchanged
305
+ 9. Refresh or replace the heuristic baseline after measured sweep, fixtures, and manual playtesting so the repo has credible baseline evidence on the repaired family.
306
+ Success bar: the heuristic should at least be competitive with, and preferably beat, random on the repaired-family task.
307
 
308
  ### Secondary Cleanup
309
 
310
+ 10. Decide whether submit should decrement `budget_remaining` for contract clarity.
311
+ 11. Import `N_FIELD_PERIODS` in [server/app.py](../server/app.py) directly from `server.contract`.
312
+ 12. Clean up or intentionally commit the current workspace-only baseline and notebook artifacts.
313
 
314
  ## Bottom Line
315