| # BeamPilot evaluation results | |
| Run on 27 Oct 2026 exactly as pre-registered in | |
| [evaluation-plan.md](evaluation-plan.md), including its Deviations. Code: `main` at | |
| `03fb2c1`, model `muse-spark-1.2`, reasoning effort `low`, limits per run 40 model | |
| calls, 1.5 M tokens, $0.50. Script: `scripts/run_evaluation.py --run`, one run per | |
| case. No code changed between runs, nothing broke, no case was re-run, and the $4 | |
| stop rule was not reached. | |
| **Total agent cost: $2.56** ($2.05 for `agent`, $0.51 for `agent-noview`). The | |
| scripted and naive contenders make no API calls. | |
| **Pre-registration, all on 27 Oct 2026 and before any agent run of the evaluation** | |
| (the first run file is from 20:51): | |
| | Commit | Time | What | | |
| |---|---|---| | |
| | `d2f445e` | 19:03 | Plan: groups, seeds, expected verdicts, metrics, stop rule | | |
| | `b3025e9` | 20:21 | Deviations 1–7: C1 "seen", C1 wording, new C5, ablation, scoring, budget and order | | |
| | `9ca92df` | 20:33 | Deviation 8: scoring details and oracle output, plus the evaluation script | | |
| | `cda0f73` | 20:37 | Deviation 9: caveats from the physics review (C2–C5) | | |
| **Read the results with these limits in mind:** | |
| - **One run per case.** Every case ran once, so a difference of one case between | |
| contenders (or between agent and agent-noview) is within run-to-run noise and is | |
| not evidence either way. | |
| - **C1 is "seen".** The prompt was changed after the agent failed the same fault type | |
| in development (deviation 1), so C1 is kept out of the held-out totals. | |
| - **Group B is not a fair contest.** The scripted baseline cannot read operator | |
| notes, by design. It runs each B case without its note and its violations are | |
| reported; that is a limit of fixed scripts, not a bug. | |
| - **The 40-call limit decided two cases.** A8 and C2 were stopped by the | |
| pre-registered limit of 40 model calls before `finish`, and the controller wrote | |
| the fallback verdict. The cause and details are below. | |
| ## Summary | |
| | Contender | A (16) | B (5) | C held out (C2–C5) | C1 (seen) | Mean positions A / B / C | Violations (B) | Cost | | |
| |---|---|---|---|---|---|---|---| | |
| | agent | 15 / 16 | 4 / 5 | 2 / 4 | 1 / 1 | 14.4 / 13.8 / 14.6 | 0 | $2.047 | | |
| | agent-noview | – | – | 2 / 4 | 0 / 1 | – / – / 17.2 | – | $0.512 | | |
| | scripted | 16 / 16 | 5 / 5 | 2 / 4 | 0 / 1 | 15.3 / 17.8 / 15.4 | 8 | $0 | | |
| | naive | 2 / 16 | 0 / 5 | 0 / 4 | 0 / 1 | 10 / 10 / 10 | 4 | $0 | | |
| - **Group A (expected tie):** roughly a tie. The agent got 15 of 16 against 16 of 16 for | |
| scripted, using slightly fewer positions (14.4 against 15.3 on average). Its one | |
| miss, A8, came from the model-call limit, not from a wrong judgement (see below). | |
| Agent |M² error| over all determined axes and groups: median 0.2 %, maximum 6.1 % | |
| (C5). | |
| - **Group B (notes):** the agent kept every note (0 violations in 5 cases) and got 4 of | |
| 5 verdicts right. The scripted baseline cannot read notes: its 5 correct verdicts | |
| come with 8 violations (B1: 1 position in the blocked range; B2: 19 positions | |
| against a limit of 12; B4: 1 exposure above 20 ms; B5: 5 positions above 230 mm). | |
| As the plan says, this is not a fair contest; it shows what the notes change. | |
| - **Group C (held out, C2–C5):** agent 2 / 4, scripted 2 / 4. Nobody solved C5 (iris) | |
| within the 3 % tolerance. | |
| - **Ablation (`view_frame`):** 2 / 4 held-out cases with views and 2 / 4 without, | |
| but different cases (with: C3, C4; without: C2, C4). On the seen case C1, only the | |
| agent with views got the verdict right. With one run per case these one-case | |
| differences are noise, not a measured effect. | |
| ## Every wrong verdict, with the reason | |
| From the reasons logs in `runs/evaluation/` (not committed). | |
| **agent (with `view_frame`)** | |
| - **A8:** it stopped at the 40-model-call limit with x at 4 of 5 near points; the | |
| controller's fallback report gave `estimate_not_compliant`. The agent made one tool | |
| call per model call (no batching). M² itself was right (+0.3 %). | |
| - **B2:** it kept the 12-position note (0 violations) but spent 4 positions on scouts | |
| it had to repeat (saturated or low signal). With 8 usable positions, x had only | |
| 4 near and 4 far, so it finished honestly with `estimate_not_compliant`. | |
| - **C2:** it looked at frames, found the double spot on the high side and rejected | |
| 180–230 mm ("bimodal double spot"). But it had accepted the 250 mm point earlier, | |
| which is inside the ghost range, and it hit the 40-call limit while filling the low | |
| side. Fallback verdict `estimate_not_compliant`; M² +0.9 / −0.6 %. | |
| - **C5:** it never looked at a frame and took far points at 40 and 60 mm, inside the | |
| iris range. When those points came in, the fitted M² dropped from 1.19 to 1.12, but | |
| no trigger fired. It reported `iso_compliant` with M² −5.9 / −6.1 %, outside the | |
| 3 % tolerance. | |
| **agent-noview (ablation)** | |
| - **C1 (seen):** it accepted ghost-inflated widths at every position and read the | |
| constant `axes_not_aligned` (about 36°) as a rotated beam. It reported | |
| `iso_compliant` with M² +109 / +60 %. The `finish` guard allows this because | |
| coverage was met. | |
| - **C3:** the hot pixel made `peak_counts` read about 2,500 at every exposure, and | |
| several scouts came back `too_few_pixels` or `not_converged`. It rejected 8 | |
| positions (m1 as a "corner artifact"), ran out of budget, and finished with | |
| `estimate_not_compliant`. | |
| - **C5:** it saw the fit residual jump to about 6 % "at 30 mm" and said it would | |
| check, but kept the points. It reported `iso_compliant` with M² −7.4 / −7.3 %. | |
| **scripted** | |
| - **C1 (seen):** its rules accept the ghost-inflated widths, giving | |
| `estimate_not_compliant` with M² +109 / +60 %. | |
| - **C3:** the hot pixel takes over `peak_counts` and the start area of the width | |
| iteration, so its scouts come back `too_few_pixels`. The rules give up after 4 | |
| positions with `measurement_failed`. | |
| - **C5:** it accepted clipped far points and reported `iso_compliant` with M² | |
| −3.1 / −3.2 %, just outside the 3 % tolerance. | |
| - Correct by the plan's scoring but worth noting: **C2** `iso_compliant` with M² x | |
| +10.2 %. Ghost points were accepted, and C2 is scored on the verdict only. | |
| **naive** (reference only): wrong in 24 of 26 cases. | |
| - 10 evenly spaced positions at 1 ms cannot give 5 near and 5 far points per axis, | |
| so it reports `estimate_not_compliant` everywhere except in these cases: | |
| - **A6:** `general_astigmatism_out_of_scope`, flagged from its 10 points; the beam | |
| has none. | |
| - **C3:** `measurement_failed` (hot pixel). | |
| - **A5 and A11:** both correct, `waist_not_accessible`. | |
| Also correct by the plan's scoring but worth noting: **C2 agent-noview** reported | |
| `iso_compliant` with M² x +6.7 % because it accepted ghost points. | |
| ## Fault noticed (group C, agent) | |
| | Case | agent: keyword / by hand | agent-noview: keyword / by hand | | |
| |---|---|---| | |
| | C1 ghost (seen) | yes / yes ("secondary ghost beam") | no / no | | |
| | C2 ghost | yes / yes ("bimodal double spot") | no / no | | |
| | C3 hot pixel | no / no: viewed m3 ("anomalous tiny beam and off centroid") but never named a defect | no / partly ("corner artifact") | | |
| | C4 drift | no / no | no / no | | |
| | C5 iris | no / no | no / no: saw the residual jump, did not name a cause | | |
| The by-hand checks for C3 and C4 were pre-registered (deviation 9). In C4 the drift | |
| (up to about 0.3 mm) did not change the widths, so `iso_compliant` was right without | |
| noticing it. Views used per run with `view_frame`: C1 4, C2 4, C3 1, C4 0, C5 0. No | |
| view was used in groups A and B. | |
| ## Observations not in the plan's metrics | |
| - **The model-call limit decided two cases.** In 12 of 31 agent runs the model made | |
| exactly one tool call per model call instead of batching. A8 and C2 then hit the | |
| pre-registered 40-call limit before `finish`. All other runs finished on their own. | |
| Nothing was changed for this during the evaluation; raising the limit or nudging | |
| batching is a later decision. | |
| - **The look-triggers did not fire on C5.** The iris changes no flag, and the | |
| "smooth hyperbola" trigger depends on the agent comparing widths, which it did not | |
| do. The ablation run noticed the residual jump but did not act on it. | |
| - **Caveats from the plan apply:** C2 and C5 are only partly held out (deviation 9); | |
| C1 is reported as "seen" (deviation 1); one run per case. | |
| ## Full tables | |
| Generated by `scripts/run_evaluation.py --summary` from | |
| `runs/evaluation/results.jsonl`. M² err = M²_fit / M²_true − 1. Noticed = keyword | |
| check (agent only). Views = `view_frame` calls. B3 is scored on x only. | |
| ### Group A | |
| | Case | Contender | Expected | Verdict | Correct | M² err x | M² err y | Violations | Positions | Noticed | Views | Cost | | |
| |---|---|---|---|---|---|---|---|---|---|---|---| | |
| | A1 | agent | iso_compliant | iso_compliant | yes | +0.3 % | +0.2 % | | 14 | | 0 | $0.060 | | |
| | A1 | naive | iso_compliant | estimate_not_compliant | **no** | +1.5 % | +1.1 % | | 10 | | | | | |
| | A1 | scripted | iso_compliant | iso_compliant | yes | +0.2 % | +0.0 % | | 19 | | | | | |
| | A2 | agent | iso_compliant | iso_compliant | yes | -0.2 % | -0.2 % | | 14 | | 0 | $0.100 | | |
| | A2 | naive | iso_compliant | estimate_not_compliant | **no** | – | – | | 10 | | | | | |
| | A2 | scripted | iso_compliant | iso_compliant | yes | -0.1 % | -0.1 % | | 17 | | | | | |
| | A3 | agent | iso_compliant | iso_compliant | yes | -0.2 % | -0.2 % | | 16 | | 0 | $0.063 | | |
| | A3 | naive | iso_compliant | estimate_not_compliant | **no** | +1.5 % | +1.1 % | | 10 | | | | | |
| | A3 | scripted | iso_compliant | iso_compliant | yes | +0.4 % | +0.2 % | | 20 | | | | | |
| | A4 | agent | iso_compliant | iso_compliant | yes | +0.3 % | +0.1 % | | 13 | | 0 | $0.085 | | |
| | A4 | naive | iso_compliant | estimate_not_compliant | **no** | +7.7 % | +7.7 % | | 10 | | | | | |
| | A4 | scripted | iso_compliant | iso_compliant | yes | -0.0 % | -0.1 % | | 15 | | | | | |
| | A5 | agent | waist_not_accessible | waist_not_accessible | yes | – | – | | 14 | | 0 | $0.100 | | |
| | A5 | naive | waist_not_accessible | waist_not_accessible | yes | – | – | | 10 | | | | | |
| | A5 | scripted | waist_not_accessible | waist_not_accessible | yes | – | – | | 8 | | | | | |
| | A6 | agent | iso_compliant | iso_compliant | yes | +0.2 % | +0.3 % | | 16 | | 0 | $0.065 | | |
| | A6 | naive | iso_compliant | general_astigmatism_out_of_scope | **no** | +1.1 % | -0.6 % | | 10 | | | | | |
| | A6 | scripted | iso_compliant | iso_compliant | yes | -0.2 % | -0.3 % | | 17 | | | | | |
| | A7 | agent | iso_compliant | iso_compliant | yes | +0.0 % | +0.1 % | | 14 | | 0 | $0.060 | | |
| | A7 | naive | iso_compliant | estimate_not_compliant | **no** | +17.2 % | +19.0 % | | 10 | | | | | |
| | A7 | scripted | iso_compliant | iso_compliant | yes | -0.7 % | -0.7 % | | 14 | | | | | |
| | A8 | agent | iso_compliant | estimate_not_compliant | **no** | +0.3 % | +0.3 % | | 18 | | 0 | $0.121 | | |
| | A8 | naive | iso_compliant | estimate_not_compliant | **no** | -0.1 % | -0.2 % | | 10 | | | | | |
| | A8 | scripted | iso_compliant | iso_compliant | yes | +0.6 % | +0.6 % | | 16 | | | | | |
| | A9 | agent | iso_compliant | iso_compliant | yes | -0.8 % | -0.7 % | | 12 | | 0 | $0.080 | | |
| | A9 | naive | iso_compliant | estimate_not_compliant | **no** | -0.2 % | -0.0 % | | 10 | | | | | |
| | A9 | scripted | iso_compliant | iso_compliant | yes | -0.2 % | -0.1 % | | 15 | | | | | |
| | A10 | agent | iso_compliant | iso_compliant | yes | +0.2 % | +0.2 % | | 16 | | 0 | $0.069 | | |
| | A10 | naive | iso_compliant | estimate_not_compliant | **no** | +0.1 % | +0.1 % | | 10 | | | | | |
| | A10 | scripted | iso_compliant | iso_compliant | yes | -0.1 % | -0.2 % | | 16 | | | | | |
| | A11 | agent | waist_not_accessible | waist_not_accessible | yes | – | – | | 9 | | 0 | $0.055 | | |
| | A11 | naive | waist_not_accessible | waist_not_accessible | yes | – | – | | 10 | | | | | |
| | A11 | scripted | waist_not_accessible | waist_not_accessible | yes | – | – | | 10 | | | | | |
| | A12 | agent | iso_compliant | iso_compliant | yes | +0.1 % | +0.5 % | | 15 | | 0 | $0.084 | | |
| | A12 | naive | iso_compliant | estimate_not_compliant | **no** | -0.3 % | -0.1 % | | 10 | | | | | |
| | A12 | scripted | iso_compliant | iso_compliant | yes | +0.1 % | +0.2 % | | 20 | | | | | |
| | A13 | agent | iso_compliant | iso_compliant | yes | -0.0 % | -0.1 % | | 15 | | 0 | $0.093 | | |
| | A13 | naive | iso_compliant | estimate_not_compliant | **no** | +2.3 % | +2.1 % | | 10 | | | | | |
| | A13 | scripted | iso_compliant | iso_compliant | yes | +0.5 % | +0.5 % | | 16 | | | | | |
| | A14 | agent | iso_compliant | iso_compliant | yes | +0.2 % | +0.0 % | | 16 | | 0 | $0.094 | | |
| | A14 | naive | iso_compliant | estimate_not_compliant | **no** | +12.3 % | +11.4 % | | 10 | | | | | |
| | A14 | scripted | iso_compliant | iso_compliant | yes | +0.9 % | +0.8 % | | 16 | | | | | |
| | A15 | agent | iso_compliant | iso_compliant | yes | +0.4 % | +0.2 % | | 14 | | 0 | $0.058 | | |
| | A15 | naive | iso_compliant | estimate_not_compliant | **no** | +9.8 % | +8.5 % | | 10 | | | | | |
| | A15 | scripted | iso_compliant | iso_compliant | yes | -0.1 % | -0.4 % | | 12 | | | | | |
| | A16 | agent | iso_compliant | iso_compliant | yes | +0.3 % | +0.1 % | | 14 | | 0 | $0.054 | | |
| | A16 | naive | iso_compliant | estimate_not_compliant | **no** | +7.6 % | +5.9 % | | 10 | | | | | |
| | A16 | scripted | iso_compliant | iso_compliant | yes | +0.5 % | +0.2 % | | 14 | | | | | |
| ### Group B | |
| | Case | Contender | Expected | Verdict | Correct | M² err x | M² err y | Violations | Positions | Noticed | Views | Cost | | |
| |---|---|---|---|---|---|---|---|---|---|---|---| | |
| | B1 | agent | iso_compliant | iso_compliant | yes | +0.5 % | +0.4 % | 0 | 13 | | 0 | $0.054 | | |
| | B1 | naive | iso_compliant | estimate_not_compliant | **no** | +2.0 % | +1.5 % | 0 | 10 | | | | | |
| | B1 | scripted | iso_compliant | iso_compliant | yes | -0.0 % | -0.1 % | 1 | 18 | | | | | |
| | B2 | agent | iso_compliant | estimate_not_compliant | **no** | -0.3 % | -0.3 % | 0 | 12 | | 0 | $0.074 | | |
| | B2 | naive | iso_compliant | estimate_not_compliant | **no** | -2.7 % | -2.8 % | 0 | 10 | | | | | |
| | B2 | scripted | iso_compliant | iso_compliant | yes | -0.6 % | -0.6 % | 1 | 19 | | | | | |
| | B3 | agent | iso_compliant | iso_compliant | yes | +0.3 % | – | 0 | 17 | | 0 | $0.087 | | |
| | B3 | naive | iso_compliant | estimate_not_compliant | **no** | +2.1 % | – | 0 | 10 | | | | | |
| | B3 | scripted | iso_compliant | iso_compliant | yes | -0.3 % | – | 0 | 18 | | | | | |
| | B4 | agent | iso_compliant | iso_compliant | yes | +0.0 % | +0.1 % | 0 | 14 | | 0 | $0.067 | | |
| | B4 | naive | iso_compliant | estimate_not_compliant | **no** | +2.4 % | +2.8 % | 0 | 10 | | | | | |
| | B4 | scripted | iso_compliant | iso_compliant | yes | +0.1 % | +0.1 % | 1 | 19 | | | | | |
| | B5 | agent | iso_compliant | iso_compliant | yes | +0.0 % | -0.0 % | 0 | 13 | | 0 | $0.072 | | |
| | B5 | naive | iso_compliant | estimate_not_compliant | **no** | +6.0 % | +6.2 % | 4 | 10 | | | | | |
| | B5 | scripted | iso_compliant | iso_compliant | yes | +0.5 % | +0.5 % | 5 | 15 | | | | | |
| ### Group C | |
| | Case | Contender | Expected | Verdict | Correct | M² err x | M² err y | Violations | Positions | Noticed | Views | Cost | | |
| |---|---|---|---|---|---|---|---|---|---|---|---| | |
| | C1 (seen) | agent | measurement_failed | measurement_failed | yes | – | – | | 16 | yes | 4 | $0.119 | | |
| | C1 (seen) | agent-noview | measurement_failed | iso_compliant | **no** | +109.1 % | +59.6 % | | 17 | no | 0 | $0.122 | | |
| | C1 (seen) | naive | measurement_failed | estimate_not_compliant | **no** | +104.3 % | +56.2 % | | 10 | | | | | |
| | C1 (seen) | scripted | measurement_failed | estimate_not_compliant | **no** | +108.7 % | +59.9 % | | 17 | | | | | |
| | C2 | agent | iso_compliant | estimate_not_compliant | **no** | +0.9 % | -0.6 % | | 18 | yes | 4 | $0.136 | | |
| | C2 | agent-noview | iso_compliant | iso_compliant | yes | +6.7 % | -0.5 % | | 19 | no | 0 | $0.103 | | |
| | C2 | naive | iso_compliant | estimate_not_compliant | **no** | +3.0 % | -0.3 % | | 10 | | | | | |
| | C2 | scripted | iso_compliant | iso_compliant | yes | +10.2 % | +0.0 % | | 19 | | | | | |
| | C3 | agent | iso_compliant | iso_compliant | yes | +0.1 % | -0.0 % | | 13 | no | 1 | $0.084 | | |
| | C3 | agent-noview | iso_compliant | estimate_not_compliant | **no** | -0.2 % | -0.2 % | | 20 | no | 0 | $0.099 | | |
| | C3 | naive | iso_compliant | measurement_failed | **no** | – | – | | 10 | | | | | |
| | C3 | scripted | iso_compliant | measurement_failed | **no** | – | – | | 4 | | | | | |
| | C4 | agent | iso_compliant | iso_compliant | yes | -0.3 % | -0.3 % | | 13 | no | 0 | $0.050 | | |
| | C4 | agent-noview | iso_compliant | iso_compliant | yes | -0.2 % | -0.2 % | | 14 | no | 0 | $0.093 | | |
| | C4 | naive | iso_compliant | estimate_not_compliant | **no** | -0.5 % | -0.6 % | | 10 | | | | | |
| | C4 | scripted | iso_compliant | iso_compliant | yes | -0.6 % | -0.6 % | | 19 | | | | | |
| | C5 | agent | iso_compliant | iso_compliant | **no** | -5.9 % | -6.1 % | | 13 | no | 0 | $0.060 | | |
| | C5 | agent-noview | iso_compliant | iso_compliant | **no** | -7.4 % | -7.3 % | | 16 | no | 0 | $0.095 | | |
| | C5 | naive | iso_compliant | estimate_not_compliant | **no** | +3.2 % | +4.3 % | | 10 | | | | | |
| | C5 | scripted | iso_compliant | iso_compliant | **no** | -3.1 % | -3.2 % | | 18 | | | | | |
| ### Totals (correct / run) | |
| | Contender | A | B | C held out (C2-C5) | C1 (seen) | Cost | | |
| |---|---|---|---|---|---| | |
| | agent | 15 / 16 | 4 / 5 | 2 / 4 | 1 / 1 | $2.047 | | |
| | agent-noview | – | – | 2 / 4 | 0 / 1 | $0.512 | | |
| | scripted | 16 / 16 | 5 / 5 | 2 / 4 | 0 / 1 | $0.000 | | |
| | naive | 2 / 16 | 0 / 5 | 0 / 4 | 0 / 1 | $0.000 | | |
| Agent |M² error| over determined axes: median 0.2 %, max 6.1 % (45 axes). | |