Commit ·
80ce5e5
1
Parent(s): 92c59c2
docs: polish verifier reward review
Browse files
docs/VERIFIER_REWARD_REVIEW_2026-03-08.md
CHANGED
|
@@ -13,10 +13,10 @@ This note reviews how well the current verifier path and reward function serve t
|
|
| 13 |
|
| 14 |
Primary review targets:
|
| 15 |
|
| 16 |
-
- [
|
| 17 |
-
- [
|
| 18 |
-
- [server/physics.py](/
|
| 19 |
-
- [server/environment.py](/
|
| 20 |
- recent history from `d58c100` through `6deaccc`
|
| 21 |
|
| 22 |
## Executive Summary
|
|
@@ -48,10 +48,9 @@ The planning docs explicitly say the environment artifact is the product, not ju
|
|
| 48 |
|
| 49 |
Relevant references:
|
| 50 |
|
| 51 |
-
- [
|
| 52 |
-
- [
|
| 53 |
-
- [
|
| 54 |
-
- [server/data/p1/README.md](/Users/suhjungdae/code/fusion-design-lab/server/data/p1/README.md#L15)
|
| 55 |
|
| 56 |
### 2. Observation legibility still conflicts with the official tolerance semantics
|
| 57 |
|
|
@@ -67,21 +66,23 @@ This is a direct usability problem for a repo whose goal is a legible, human-pla
|
|
| 67 |
|
| 68 |
Relevant references:
|
| 69 |
|
| 70 |
-
- [server/environment.py](/
|
| 71 |
-
- [server/physics.py](/
|
| 72 |
-
- [
|
| 73 |
|
| 74 |
### 3. High-fidelity best-state bookkeeping is still not trustworthy
|
| 75 |
|
| 76 |
`best_high_fidelity_score` and `best_high_fidelity_feasibility` are not true best high-fidelity values. On submit, the env can overwrite them by reevaluating `best_params`, even if the current submit is the best actual high-fidelity result.
|
| 77 |
|
|
|
|
|
|
|
| 78 |
This weakens the clean low-fi/high-fi split the contract calls for.
|
| 79 |
|
| 80 |
Relevant references:
|
| 81 |
|
| 82 |
-
- [server/environment.py](/
|
| 83 |
-
- [server/environment.py](/
|
| 84 |
-
- [
|
| 85 |
|
| 86 |
### 4. Submit is operationally expensive
|
| 87 |
|
|
@@ -91,35 +92,48 @@ One `submit` can trigger up to three high-fidelity evaluations:
|
|
| 91 |
2. initial high-fidelity reference
|
| 92 |
3. reevaluated `best_params`
|
| 93 |
|
|
|
|
|
|
|
| 94 |
That makes baseline comparison and manual validation much slower than the task surface implies.
|
| 95 |
|
| 96 |
Relevant references:
|
| 97 |
|
| 98 |
-
- [server/environment.py](/
|
| 99 |
-
- [server/environment.py](/
|
|
|
|
|
|
|
| 100 |
|
| 101 |
-
|
| 102 |
|
| 103 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
|
| 105 |
That means a failed submit can still count as a terminal high-fi result in comparison output.
|
| 106 |
|
| 107 |
-
###
|
| 108 |
|
| 109 |
-
[server/physics.py](/
|
| 110 |
|
| 111 |
-
###
|
| 112 |
|
| 113 |
`submit` does not decrement `budget_remaining`, so the terminal observation shows pre-submit budget. Since `submit` is terminal, this is not a serious episode-flow bug, but it is a contract/reporting inconsistency.
|
| 114 |
|
| 115 |
Relevant reference:
|
| 116 |
|
| 117 |
-
- [server/environment.py](/
|
| 118 |
|
| 119 |
-
###
|
| 120 |
|
| 121 |
-
- [server/app.py](/
|
| 122 |
-
- [fusion_lab/models.py](
|
| 123 |
|
| 124 |
## Findings From The Quoted External Review
|
| 125 |
|
|
@@ -135,7 +149,7 @@ Relevant reference:
|
|
| 135 |
|
| 136 |
- submit budget reporting is inconsistent, but it does not allow the agent to exceed the episode budget because submit is terminal
|
| 137 |
|
| 138 |
-
### Not
|
| 139 |
|
| 140 |
The quoted review's highest-severity claim was an `_update_best` feasibility mismatch between:
|
| 141 |
|
|
@@ -144,7 +158,16 @@ The quoted review's highest-severity claim was an `_update_best` feasibility mis
|
|
| 144 |
|
| 145 |
I did not reproduce that as a current live bug. I sampled 30 low-fidelity evaluations across the current parameter ranges and found zero mismatches between those two criteria.
|
| 146 |
|
| 147 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
|
| 149 |
## Validation Performed
|
| 150 |
|
|
@@ -166,16 +189,33 @@ So the task is not trivially solved at reset.
|
|
| 166 |
|
| 167 |
### Sampled low-fidelity trajectory
|
| 168 |
|
| 169 |
-
For seed `0`,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
|
| 171 |
This is a good sign for current `Reward V0` ordering, but it is not a substitute for the fixture/manual-playtest loop the docs require.
|
| 172 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
### Current workspace state at review time
|
| 174 |
|
| 175 |
The workspace was dirty during review:
|
| 176 |
|
| 177 |
-
- modified [training/notebooks/northflank_smoke.py](/
|
| 178 |
-
- untracked [baselines/measured_sweep.py](/
|
| 179 |
- untracked `baselines/sweep_results/`
|
| 180 |
|
| 181 |
These were treated as workspace observations, not committed `HEAD` findings.
|
|
@@ -235,6 +275,7 @@ Weaknesses:
|
|
| 235 |
|
| 236 |
- not yet validated by the playtest/fixture loop the repo requires
|
| 237 |
- some reward/reporting details still depend on unclear high-fi bookkeeping
|
|
|
|
| 238 |
|
| 239 |
## TODOs
|
| 240 |
|
|
@@ -242,12 +283,12 @@ Weaknesses:
|
|
| 242 |
|
| 243 |
1. Fix diagnostics so humans can understand official tolerance-based feasibility.
|
| 244 |
2. Fix true high-fidelity best-state bookkeeping.
|
| 245 |
-
3. Update [baselines/compare.py](/
|
| 246 |
4. Reduce submit-time high-fidelity reevaluation cost.
|
| 247 |
|
| 248 |
### Validation Priority
|
| 249 |
|
| 250 |
-
5. Add tracked `P1` fixtures under [server/data/p1/](/
|
| 251 |
6. Run fixture sanity checks for:
|
| 252 |
- good or near-boundary cases
|
| 253 |
- clearly bad cases
|
|
@@ -261,12 +302,14 @@ Weaknesses:
|
|
| 261 |
8. Either:
|
| 262 |
- document the first real `Reward V0 -> V1` pathology/fix
|
| 263 |
- or explicitly record that `Reward V0` survived playtesting unchanged
|
|
|
|
|
|
|
| 264 |
|
| 265 |
### Secondary Cleanup
|
| 266 |
|
| 267 |
-
|
| 268 |
-
|
| 269 |
-
|
| 270 |
|
| 271 |
## Bottom Line
|
| 272 |
|
|
|
|
| 13 |
|
| 14 |
Primary review targets:
|
| 15 |
|
| 16 |
+
- [FUSION_DESIGN_LAB_PLAN_V2.md](./FUSION_DESIGN_LAB_PLAN_V2.md)
|
| 17 |
+
- [P1_ENV_CONTRACT_V1.md](./P1_ENV_CONTRACT_V1.md)
|
| 18 |
+
- [server/physics.py](../server/physics.py)
|
| 19 |
+
- [server/environment.py](../server/environment.py)
|
| 20 |
- recent history from `d58c100` through `6deaccc`
|
| 21 |
|
| 22 |
## Executive Summary
|
|
|
|
| 48 |
|
| 49 |
Relevant references:
|
| 50 |
|
| 51 |
+
- [FUSION_DESIGN_LAB_PLAN_V2.md](./FUSION_DESIGN_LAB_PLAN_V2.md)
|
| 52 |
+
- [FUSION_DELIVERABLES_MAP.md](./FUSION_DELIVERABLES_MAP.md)
|
| 53 |
+
- [server/data/p1/README.md](../server/data/p1/README.md)
|
|
|
|
| 54 |
|
| 55 |
### 2. Observation legibility still conflicts with the official tolerance semantics
|
| 56 |
|
|
|
|
| 66 |
|
| 67 |
Relevant references:
|
| 68 |
|
| 69 |
+
- [server/environment.py](../server/environment.py) in `_build_observation(...)`
|
| 70 |
+
- [server/physics.py](../server/physics.py) in `_to_evaluation_metrics(...)`
|
| 71 |
+
- [P1_PARAMETERIZATION_DEEPDIVE.md](./P1_PARAMETERIZATION_DEEPDIVE.md)
|
| 72 |
|
| 73 |
### 3. High-fidelity best-state bookkeeping is still not trustworthy
|
| 74 |
|
| 75 |
`best_high_fidelity_score` and `best_high_fidelity_feasibility` are not true best high-fidelity values. On submit, the env can overwrite them by reevaluating `best_params`, even if the current submit is the best actual high-fidelity result.
|
| 76 |
|
| 77 |
+
The worst failure mode is sharper than simple bookkeeping drift: if reevaluating `best_params` crashes VMEC, the env can store failure-sentinel data under `best_high_fidelity_*`, including `p1_score=0.0`, while still labeling that state as "best."
|
| 78 |
+
|
| 79 |
This weakens the clean low-fi/high-fi split the contract calls for.
|
| 80 |
|
| 81 |
Relevant references:
|
| 82 |
|
| 83 |
+
- [server/environment.py](../server/environment.py) in `_refresh_best_high_fidelity_metrics(...)`
|
| 84 |
+
- [server/environment.py](../server/environment.py) in `_summary_submit(...)`
|
| 85 |
+
- [P1_ENV_CONTRACT_V1.md](./P1_ENV_CONTRACT_V1.md)
|
| 86 |
|
| 87 |
### 4. Submit is operationally expensive
|
| 88 |
|
|
|
|
| 92 |
2. initial high-fidelity reference
|
| 93 |
3. reevaluated `best_params`
|
| 94 |
|
| 95 |
+
The initial high-fidelity reference cache is only per episode because it lives on `State`, which is recreated on every `reset()`. In practice, repeated episodes still repay that cost.
|
| 96 |
+
|
| 97 |
That makes baseline comparison and manual validation much slower than the task surface implies.
|
| 98 |
|
| 99 |
Relevant references:
|
| 100 |
|
| 101 |
+
- [server/environment.py](../server/environment.py) in `_handle_submit(...)`
|
| 102 |
+
- [server/environment.py](../server/environment.py) in `_initial_high_fidelity_score(...)`
|
| 103 |
+
|
| 104 |
+
### 5. Heuristic baseline evidence is stale and currently unconvincing
|
| 105 |
|
| 106 |
+
The current heuristic is not validated on the repaired 4-knob family and is already documented in repo status notes as underperforming random on the real verifier path.
|
| 107 |
|
| 108 |
+
This is not a verifier bug, but it is a serious artifact-quality problem for a repo that needs credible baseline evidence.
|
| 109 |
+
|
| 110 |
+
Relevant references:
|
| 111 |
+
|
| 112 |
+
- [README.md](../README.md)
|
| 113 |
+
- [TODO.md](../TODO.md)
|
| 114 |
+
|
| 115 |
+
### 6. Baseline comparison does not reject failed high-fidelity submits
|
| 116 |
+
|
| 117 |
+
[baselines/compare.py](../baselines/compare.py) in `_require_submit_fidelity(...)` only checks that the final step used high-fidelity evaluation. It does not check whether that high-fidelity submit succeeded.
|
| 118 |
|
| 119 |
That means a failed submit can still count as a terminal high-fi result in comparison output.
|
| 120 |
|
| 121 |
+
### 7. Verifier failure normalization is incomplete
|
| 122 |
|
| 123 |
+
[server/physics.py](../server/physics.py) in `evaluate_boundary(...)` converts `RuntimeError` into explicit failure metrics, but other exception types can still crash the env instead of producing documented failure observations.
|
| 124 |
|
| 125 |
+
### 8. Submit budget reporting is inconsistent
|
| 126 |
|
| 127 |
`submit` does not decrement `budget_remaining`, so the terminal observation shows pre-submit budget. Since `submit` is terminal, this is not a serious episode-flow bug, but it is a contract/reporting inconsistency.
|
| 128 |
|
| 129 |
Relevant reference:
|
| 130 |
|
| 131 |
+
- [server/environment.py](../server/environment.py) in `_handle_submit(...)`
|
| 132 |
|
| 133 |
+
### 9. Minor maintenance smells
|
| 134 |
|
| 135 |
+
- [server/app.py](../server/app.py) imports `N_FIELD_PERIODS` indirectly via `server.environment` instead of directly from `server.contract`
|
| 136 |
+
- [fusion_lab/models.py](../fusion_lab/models.py) has misleading defaults if used outside the environment
|
| 137 |
|
| 138 |
## Findings From The Quoted External Review
|
| 139 |
|
|
|
|
| 149 |
|
| 150 |
- submit budget reporting is inconsistent, but it does not allow the agent to exceed the episode budget because submit is terminal
|
| 151 |
|
| 152 |
+
### Fragile Coupling, Not A Reproduced Live Bug
|
| 153 |
|
| 154 |
The quoted review's highest-severity claim was an `_update_best` feasibility mismatch between:
|
| 155 |
|
|
|
|
| 158 |
|
| 159 |
I did not reproduce that as a current live bug. I sampled 30 low-fidelity evaluations across the current parameter ranges and found zero mismatches between those two criteria.
|
| 160 |
|
| 161 |
+
But the current code still represents a fragile coupling: `_update_best(...)` uses one concept through `constraints_satisfied` for `current`, and a separate hardcoded `FEASIBILITY_TOLERANCE` comparison for `best`. Those happen to agree today because current `constellaration` semantics line up with `compute_feasibility() <= 0.01`.
|
| 162 |
+
|
| 163 |
+
That means this is best described as:
|
| 164 |
+
|
| 165 |
+
- not a reproduced live bug on current `HEAD`
|
| 166 |
+
- still a maintainability risk if upstream tolerance semantics ever drift
|
| 167 |
+
|
| 168 |
+
Relevant reference:
|
| 169 |
+
|
| 170 |
+
- [server/environment.py](../server/environment.py) in `_update_best(...)`
|
| 171 |
|
| 172 |
## Validation Performed
|
| 173 |
|
|
|
|
| 189 |
|
| 190 |
### Sampled low-fidelity trajectory
|
| 191 |
|
| 192 |
+
For seed `0`, the following low-fidelity sequence reached feasibility within 5 steps:
|
| 193 |
+
|
| 194 |
+
1. `triangularity_scale increase small`
|
| 195 |
+
2. `triangularity_scale increase small`
|
| 196 |
+
3. `triangularity_scale increase small`
|
| 197 |
+
4. `rotational_transform increase small`
|
| 198 |
+
5. `rotational_transform increase small`
|
| 199 |
+
|
| 200 |
+
The feasibility-crossing step received a strong positive reward (`+3.1054`).
|
| 201 |
|
| 202 |
This is a good sign for current `Reward V0` ordering, but it is not a substitute for the fixture/manual-playtest loop the docs require.
|
| 203 |
|
| 204 |
+
One additional playtest note: reward stacking should be checked explicitly. A single step can combine:
|
| 205 |
+
|
| 206 |
+
- recovery bonus `+1.0`
|
| 207 |
+
- feasibility transition bonus `+3.0`
|
| 208 |
+
- shaping delta
|
| 209 |
+
- non-submit step cost `-0.1`
|
| 210 |
+
|
| 211 |
+
That is not obviously wrong, but it can produce a `+4.0+` step and should be verified during manual playtesting.
|
| 212 |
+
|
| 213 |
### Current workspace state at review time
|
| 214 |
|
| 215 |
The workspace was dirty during review:
|
| 216 |
|
| 217 |
+
- modified [training/notebooks/northflank_smoke.py](../training/notebooks/northflank_smoke.py)
|
| 218 |
+
- untracked [baselines/measured_sweep.py](../baselines/measured_sweep.py)
|
| 219 |
- untracked `baselines/sweep_results/`
|
| 220 |
|
| 221 |
These were treated as workspace observations, not committed `HEAD` findings.
|
|
|
|
| 275 |
|
| 276 |
- not yet validated by the playtest/fixture loop the repo requires
|
| 277 |
- some reward/reporting details still depend on unclear high-fi bookkeeping
|
| 278 |
+
- reward stacking is still under-documented and should be checked during playtesting
|
| 279 |
|
| 280 |
## TODOs
|
| 281 |
|
|
|
|
| 283 |
|
| 284 |
1. Fix diagnostics so humans can understand official tolerance-based feasibility.
|
| 285 |
2. Fix true high-fidelity best-state bookkeeping.
|
| 286 |
+
3. Update [baselines/compare.py](../baselines/compare.py) to reject or explicitly flag failed high-fidelity submits.
|
| 287 |
4. Reduce submit-time high-fidelity reevaluation cost.
|
| 288 |
|
| 289 |
### Validation Priority
|
| 290 |
|
| 291 |
+
5. Add tracked `P1` fixtures under [server/data/p1/](../server/data/p1/).
|
| 292 |
6. Run fixture sanity checks for:
|
| 293 |
- good or near-boundary cases
|
| 294 |
- clearly bad cases
|
|
|
|
| 302 |
8. Either:
|
| 303 |
- document the first real `Reward V0 -> V1` pathology/fix
|
| 304 |
- or explicitly record that `Reward V0` survived playtesting unchanged
|
| 305 |
+
9. Refresh or replace the heuristic baseline after measured sweep, fixtures, and manual playtesting so the repo has credible baseline evidence on the repaired family.
|
| 306 |
+
Success bar: the heuristic should at least be competitive with, and preferably beat, random on the repaired-family task.
|
| 307 |
|
| 308 |
### Secondary Cleanup
|
| 309 |
|
| 310 |
+
10. Decide whether submit should decrement `budget_remaining` for contract clarity.
|
| 311 |
+
11. Import `N_FIELD_PERIODS` in [server/app.py](../server/app.py) directly from `server.contract`.
|
| 312 |
+
12. Clean up or intentionally commit the current workspace-only baseline and notebook artifacts.
|
| 313 |
|
| 314 |
## Bottom Line
|
| 315 |
|