.cache/known_methods_snapshot.json CHANGED
@@ -125,6 +125,18 @@
125
  "levenberg-marquardt",
126
  "gradient_descent"
127
  ]
 
 
 
 
 
 
 
 
 
 
 
 
128
  }
129
  },
130
  "observed_methods": [
 
125
  "levenberg-marquardt",
126
  "gradient_descent"
127
  ]
128
+ },
129
+ "gboed": {
130
+ "abbreviation": "GBOED",
131
+ "Method": "Goal-oriented Bayesian Optimal Experimental Design",
132
+ "parallelism": "parallel-interacting",
133
+ "update_type": "general",
134
+ "method_goal": "uq",
135
+ "emulator_use": "within-optimize",
136
+ "aliases": [
137
+ "gboed",
138
+ "boed"
139
+ ]
140
  }
141
  },
142
  "observed_methods": [
.claude/skills/leaderboard-manager/references/manage-display.md CHANGED
@@ -25,9 +25,10 @@ next time someone edits the file.
25
  whatever `selected_benchmark` is currently chosen.
26
  3. **Controls — always visible**, not collapsed. Look for `st.subheader("Scoring &
27
  Target Controls")`: target-level radio, scoring-mode radio (+ blend-weight slider
28
- when "Custom Blend" is selected), and the "Methods to display in charts" multiselect.
29
- These used to be wrapped in `st.expander(..., expanded=False)`; that was removed so
30
- users don't have to open a dropdown to see or change the target level.
 
31
  4. **Charts** — main "Mean Forward Model Runs vs Ensemble Size" chart, then (UQ only)
32
  the "Mean Iterations for Coverage" chart, then (if `show_failure_panel=True`) the
33
  failure-rate bar chart.
@@ -130,6 +131,28 @@ non-blend mode. It lives in the always-visible controls section, not behind an e
130
 
131
  ---
132
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
133
  ## Leaderboard line chart
134
 
135
  The chart that appears above the table ("Mean Forward Model Runs vs Ensemble Size") is
@@ -305,8 +328,8 @@ This encodes two *separate* requirements that are easy to conflate:
305
 
306
  When calling `_method_color()`, compute `present_abbrevs` from the actual DataFrame(s)
307
  plotted in that chart section (e.g. `all_ens_combos["abbreviation"].unique()`), **not**
308
- from the global "Methods to display in charts" multiselect. A method can be selected in
309
- the multiselect but have no data for the current benchmark/target combination — including
310
  it in the legend anyway would be misleading. Each chart section (main chart, UQ iters
311
  chart, failure-rate chart) computes and passes its own `present_abbrevs`, since the set
312
  of methods with data can differ section to section (e.g. a method might have forward-run
@@ -361,7 +384,7 @@ new Altair chart in the app:
361
  `leaderboard_df`-emptiness gate (main chart) or have its own independent emptiness
362
  check (UQ iters chart, failure panel) — see the gotcha under "Page layout" above.
363
  4. If the chart colors by `abbreviation`: use `_method_color(present_abbrevs)` computed
364
- from that section's actual plotted data, not the global multiselect.
365
  5. Restart or clear Streamlit cache to pick up Python changes (code changes take effect on
366
  restart; cache only matters for data changes).
367
  6. Smoke-test with a benchmark/target combination where every run fails (all-DNF) to
 
25
  whatever `selected_benchmark` is currently chosen.
26
  3. **Controls — always visible**, not collapsed. Look for `st.subheader("Scoring &
27
  Target Controls")`: target-level radio, scoring-mode radio (+ blend-weight slider
28
+ when "Custom Blend" is selected), and the "Methods to display in charts" pills
29
+ selector (see "Methods selector" below). These used to be wrapped in
30
+ `st.expander(..., expanded=False)`; that was removed so users don't have to open a
31
+ dropdown to see or change the target level.
32
  4. **Charts** — main "Mean Forward Model Runs vs Ensemble Size" chart, then (UQ only)
33
  the "Mean Iterations for Coverage" chart, then (if `show_failure_panel=True`) the
34
  failure-rate bar chart.
 
131
 
132
  ---
133
 
134
+ ## Methods selector
135
+
136
+ The "Methods to display in charts" control (grep for that string in `leaderboard.py`) uses
137
+ `st.pills(..., selection_mode="multi")`, not `st.multiselect` or `st.selectbox`. Prefer
138
+ `st.pills` in multi mode whenever the goal is "let users see and toggle all options at a
139
+ glance" — every option renders as a clickable button, so nothing is hidden behind a
140
+ dropdown the way `st.multiselect`'s collapsed chip list is. `selection_mode` defaults to
141
+ `"single"`, so it must be passed explicitly to allow more than one method selected at once;
142
+ in multi mode `st.pills` returns a list, same shape as `st.multiselect`. It was added in
143
+ Streamlit 1.40 — the project's pinned `streamlit` (currently 1.57.0 in the `.venv`) already
144
+ covers it.
145
+
146
+ It reads/writes the same `key=k_methods` session-state slot the old multiselect used, so
147
+ the surrounding logic is unaffected by the widget swap: the default-selection (top 5 by
148
+ rank) and stale-selection-pruning code sets `st.session_state[k_methods]` *before* the
149
+ widget call (search for `st.session_state[k_methods] = valid_saved if valid_saved else
150
+ default_methods`), and the widget itself is called with `key=k_methods` and no `default=`
151
+ argument. If you add a new always-visible, multi-choice control elsewhere in the app,
152
+ follow this same pattern rather than reaching for `st.multiselect` by default.
153
+
154
+ ---
155
+
156
  ## Leaderboard line chart
157
 
158
  The chart that appears above the table ("Mean Forward Model Runs vs Ensemble Size") is
 
328
 
329
  When calling `_method_color()`, compute `present_abbrevs` from the actual DataFrame(s)
330
  plotted in that chart section (e.g. `all_ens_combos["abbreviation"].unique()`), **not**
331
+ from the global "Methods to display in charts" pills selector. A method can be selected
332
+ there but have no data for the current benchmark/target combination — including
333
  it in the legend anyway would be misleading. Each chart section (main chart, UQ iters
334
  chart, failure-rate chart) computes and passes its own `present_abbrevs`, since the set
335
  of methods with data can differ section to section (e.g. a method might have forward-run
 
384
  `leaderboard_df`-emptiness gate (main chart) or have its own independent emptiness
385
  check (UQ iters chart, failure panel) — see the gotcha under "Page layout" above.
386
  4. If the chart colors by `abbreviation`: use `_method_color(present_abbrevs)` computed
387
+ from that section's actual plotted data, not the global pills selector.
388
  5. Restart or clear Streamlit cache to pick up Python changes (code changes take effect on
389
  restart; cache only matters for data changes).
390
  6. Smoke-test with a benchmark/target combination where every run fails (all-DNF) to
CLAUDE.md CHANGED
@@ -61,21 +61,11 @@ calibration_benchmark/
61
  │ ├── *.nc Kalman results (TEKI, ETKI, IEKF)
62
  │ ├── bayesian/ ABC + HM results
63
  │ ├── UKI_results/ UKI results
64
- │ ├── adam_results/ ADAM gradient optimizer results
65
- │ │ filenames follow `leaderboard_adam_<benchmark-tag>_<date>.nc`;
66
- │ │ benchmark tags: `l63` → L63, `l96_const-force` → L96,
67
- │ │ `l96_vec-force` → L96_SPATIAL_FORCING (no L96_NN_FORCING yet)
68
- │ ├── levenberg_marquardt_results/ LM gradient optimizer results
69
- │ │ filenames follow `leaderboard_lm_<benchmark-tag>_<date>.nc`;
70
- │ │ same benchmark tags as adam_results/ (no L96_NN_FORCING yet)
71
  │ ├── ces-eki-dmc_results/ CES-EKI-DMC ensemble-results files (UQ source)
72
- │ │ filenames follow `ces-eki-dmc_<benchmark-tag>_ensemble_results_<date>[_minimal].nc`;
73
- │ │ benchmark tags: `l63` → L63, `l96` (no suffix) → L96,
74
- │ │ `l96_nn_forcing` → L96_NN_FORCING, `l96_spatial_forcing` → L96_SPATIAL_FORCING
75
  │ └── gnki-uq_results/ IEKF (a.k.a GNKI) ensemble-results files (UQ source)
76
- │ filenames follow `leaderboard_gnki_<benchmark-tag>_<date>[_minimal].nc`;
77
- │ benchmark tags: `l63` → L63, `l96_const-force` → L96,
78
- │ `l96_flux-force` → L96_NN_FORCING, `l96_vec-force` → L96_SPATIAL_FORCING
79
  ├── examples/ lorenz_demo.ipynb + lorenz.py (reference, not app code)
80
  ├── From_Rob_6-10/ Raw collaborator data drop — NOT read by the app
81
  ├── .cache/ known_methods_snapshot.json (auto-generated, safe to delete)
@@ -191,6 +181,7 @@ A quick taxonomy tree is rendered on the home page (`src/streamlit_app.py`).
191
  | `ces-eki-dmc` | CES-EKI-DMC | parallel-interacting | kalman | uq | after-optimize | — |
192
  | `adam` | ADAM | serial | gradient | optimization | none | — |
193
  | `lm` | LM | serial | gradient | optimization | none | gradient_descent |
 
194
 
195
  ## How to extend
196
 
 
61
  │ ├── *.nc Kalman results (TEKI, ETKI, IEKF)
62
  │ ├── bayesian/ ABC + HM results
63
  │ ├── UKI_results/ UKI results
64
+ │ ├── adam_results/ ADAM gradient optimizer results (per-benchmark files)
65
+ │ ├── levenberg_marquardt_results/ LM gradient optimizer results (per-benchmark files)
 
 
 
 
 
66
  │ ├── ces-eki-dmc_results/ CES-EKI-DMC ensemble-results files (UQ source)
67
+ │ ├── gboed-uq_results/ GBOED ensemble-results files (UQ source)
 
 
68
  │ └── gnki-uq_results/ IEKF (a.k.a GNKI) ensemble-results files (UQ source)
 
 
 
69
  ├── examples/ lorenz_demo.ipynb + lorenz.py (reference, not app code)
70
  ├── From_Rob_6-10/ Raw collaborator data drop — NOT read by the app
71
  ├── .cache/ known_methods_snapshot.json (auto-generated, safe to delete)
 
181
  | `ces-eki-dmc` | CES-EKI-DMC | parallel-interacting | kalman | uq | after-optimize | — |
182
  | `adam` | ADAM | serial | gradient | optimization | none | — |
183
  | `lm` | LM | serial | gradient | optimization | none | gradient_descent |
184
+ | `gboed` | GBOED | parallel-interacting | general | uq | within-optimize | boed |
185
 
186
  ## How to extend
187
 
data/gboed-uq_results/boed_l63_ensemble_results_2026-07-30_minimal.nc ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52616c824f068f6de1eb9ccf67f56a1000b56d1cd54c911daa817df4b600e26b
3
+ size 673057
data/gboed-uq_results/boed_l96_const-force_2026-07-30_minimal.nc ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84062bc4afda563eba840efcf06c70143037660e99f258d6ecba84538a308b78
3
+ size 809857
data/gboed-uq_results/boed_l96_vec-force_2026-08-07_minimal.nc ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d1a3cd33db2cef7d81e31cf0a679e93955f3c851153eefbf2c719061b60330b9
3
+ size 618337
src/common/leaderboard.py CHANGED
@@ -58,13 +58,20 @@ _UPDATE_TYPE_ORDER: dict[str, int] = {
58
  "general": 2,
59
  }
60
 
 
 
 
 
 
 
 
61
 
62
  def _render_suitability_table(
63
  store: pd.DataFrame,
64
  target_col: str,
65
  suitability_target: float,
66
  benchmark_dims: dict[str, tuple[int, int, int]] | None = None,
67
- failure_threshold: float = 20.0,
68
  ratio_threshold: float = 3.0,
69
  ) -> None:
70
  """Render the method × benchmark suitability grid above the leaderboard controls."""
@@ -279,13 +286,6 @@ def render_leaderboard(
279
  fwdruns_weight = current_fwdruns_weight_percent / 100.0
280
  ensemble_weight = 1.0 - fwdruns_weight
281
 
282
- # Available methods for the current benchmark selection; used to populate the
283
- # multiselect and to prune any stale saved selections when the benchmark changes.
284
- available_methods = sorted(filtered["abbreviation"].dropna().unique().tolist())
285
- saved_methods = st.session_state.get(k_methods, available_methods)
286
- valid_saved = [m for m in saved_methods if m in available_methods]
287
- st.session_state[k_methods] = valid_saved if valid_saved else available_methods
288
-
289
  def build_scored_table(input_df: pd.DataFrame, add_rank: bool = True) -> pd.DataFrame:
290
  ranking_source = input_df[input_df[target_str_col] == selected_target]
291
  if ranking_source.empty:
@@ -421,6 +421,36 @@ def render_leaderboard(
421
 
422
  leaderboard_df = build_scored_table(filtered, add_rank=True)
423
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
424
  # Attach the four taxonomy tags (constant per abbreviation, so this is a
425
  # plain lookup merge rather than a groupby key threaded through every
426
  # aggregation above).
@@ -485,10 +515,12 @@ def render_leaderboard(
485
  ),
486
  )
487
 
488
- st.multiselect(
489
  "Methods to display in charts",
490
  options=available_methods,
 
491
  key=k_methods,
 
492
  )
493
 
494
  selected_methods = st.session_state.get(k_methods, available_methods)
@@ -498,6 +530,52 @@ def render_leaderboard(
498
  st.divider()
499
 
500
  # --- Charts (rendered above the leaderboard table, below the controls) ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
501
  if not leaderboard_df.empty:
502
  st.subheader("Mean Forward Model Runs vs Ensemble Size")
503
  chart_source = filtered[filtered[target_str_col] == selected_target]
@@ -589,6 +667,72 @@ def render_leaderboard(
589
 
590
  if chart_layers:
591
  st.altair_chart(alt.layer(*chart_layers), use_container_width=True)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
592
 
593
  # Mean-iterations-for-coverage section (UQ only, when budget_store provided)
594
  if budget_store is not None and not budget_store.empty:
@@ -664,49 +808,6 @@ def render_leaderboard(
664
  if iters_layers:
665
  st.altair_chart(alt.layer(*iters_layers), use_container_width=True)
666
 
667
- # Failure panel — rendered regardless of whether the scored table has rows
668
- if show_failure_panel:
669
- failure_source = filtered[filtered[target_str_col] == selected_target]
670
- failure_source = failure_source[failure_source["abbreviation"].isin(selected_methods)]
671
- if not failure_source.empty:
672
- failure_df = failure_source.groupby(
673
- ["abbreviation", "ensemble_size"], as_index=False
674
- ).agg(mean_failure_rate=("failure_rate", "mean"))
675
- failure_df = failure_df.sort_values("ensemble_size")
676
-
677
- st.subheader(f"Failure Rate of Hitting Target {selected_target}")
678
- ens_ticks_fail = sorted(failure_df["ensemble_size"].unique().tolist())
679
- failure_chart = (
680
- alt.Chart(failure_df)
681
- .mark_bar()
682
- .encode(
683
- x=alt.X(
684
- "ensemble_size:O",
685
- title="Ensemble Size",
686
- sort=[str(e) for e in ens_ticks_fail],
687
- axis=alt.Axis(labelAngle=0),
688
- ),
689
- xOffset=alt.XOffset("abbreviation:N"),
690
- y=alt.Y(
691
- "mean_failure_rate:Q",
692
- title="Failure Rate (%)",
693
- scale=alt.Scale(domain=[0, 100]),
694
- ),
695
- color=_method_color(failure_df["abbreviation"].unique().tolist()),
696
- tooltip=[
697
- alt.Tooltip("abbreviation:N", title="Method"),
698
- alt.Tooltip("ensemble_size:O", title="Ensemble Size"),
699
- alt.Tooltip("mean_failure_rate:Q", format=".1f", title="Failure Rate (%)"),
700
- ],
701
- )
702
- )
703
- ceiling_line = (
704
- alt.Chart(alt.Data(values=[{}]))
705
- .mark_rule(color="grey", strokeDash=[4, 4])
706
- .encode(y=alt.datum(100))
707
- )
708
- st.altair_chart(failure_chart + ceiling_line, use_container_width=True)
709
-
710
  st.divider()
711
 
712
  # --- Leaderboard table (rendered below the charts) ---
 
58
  "general": 2,
59
  }
60
 
61
+ # Failure-rate threshold (%) below which a method's runs at a given target are
62
+ # considered reliable. Shared by the suitability table and the default
63
+ # "Methods to display in charts" selection so both surfaces agree on what
64
+ # counts as trustworthy — a method that only reaches a fast Score on the rare
65
+ # seed that didn't fail shouldn't outrank a slower method that never fails.
66
+ _RELIABLE_FAILURE_THRESHOLD = 20.0
67
+
68
 
69
  def _render_suitability_table(
70
  store: pd.DataFrame,
71
  target_col: str,
72
  suitability_target: float,
73
  benchmark_dims: dict[str, tuple[int, int, int]] | None = None,
74
+ failure_threshold: float = _RELIABLE_FAILURE_THRESHOLD,
75
  ratio_threshold: float = 3.0,
76
  ) -> None:
77
  """Render the method × benchmark suitability grid above the leaderboard controls."""
 
286
  fwdruns_weight = current_fwdruns_weight_percent / 100.0
287
  ensemble_weight = 1.0 - fwdruns_weight
288
 
 
 
 
 
 
 
 
289
  def build_scored_table(input_df: pd.DataFrame, add_rank: bool = True) -> pd.DataFrame:
290
  ranking_source = input_df[input_df[target_str_col] == selected_target]
291
  if ranking_source.empty:
 
421
 
422
  leaderboard_df = build_scored_table(filtered, add_rank=True)
423
 
424
+ # Methods available for the current benchmark; used to populate the pills
425
+ # selector and to prune stale saved selections when the benchmark changes. The
426
+ # default selection (first load, or once a prior selection becomes entirely
427
+ # invalid) is the top 5 methods, not every method — the "Methods to display in
428
+ # charts" pills selector lets a user opt into seeing more.
429
+ #
430
+ # Ranked primarily by reliability (failure rate below _RELIABLE_FAILURE_THRESHOLD)
431
+ # and only then by performance Rank within each reliability group. Ranking by
432
+ # Rank/Score alone would let a method that only succeeds on a rare seed (and is
433
+ # fast on that seed) outrank a method that reliably succeeds but is slower —
434
+ # e.g. on the UQ leaderboard, a mostly-failing method can post a great mean
435
+ # budget from its few converged seeds while a method with zero failures scores
436
+ # worse and gets dropped from the default view.
437
+ available_methods = sorted(filtered["abbreviation"].dropna().unique().tolist())
438
+ if not leaderboard_df.empty and leaderboard_df["Rank"].notna().any():
439
+ ranked_methods = leaderboard_df.dropna(subset=["Rank"]).copy()
440
+ ranked_methods["_unreliable"] = (
441
+ ranked_methods["Mean Failure Rate (%)"].fillna(0.0) >= _RELIABLE_FAILURE_THRESHOLD
442
+ )
443
+ default_methods = (
444
+ ranked_methods.sort_values(["_unreliable", "Rank"])["abbreviation"].head(5).tolist()
445
+ )
446
+ else:
447
+ # All runs failed at this target (or no data) — show every method so the
448
+ # all-DNF warning/chart-suppression path still has something to work with.
449
+ default_methods = available_methods
450
+ saved_methods = st.session_state.get(k_methods, default_methods)
451
+ valid_saved = [m for m in saved_methods if m in available_methods]
452
+ st.session_state[k_methods] = valid_saved if valid_saved else default_methods
453
+
454
  # Attach the four taxonomy tags (constant per abbreviation, so this is a
455
  # plain lookup merge rather than a groupby key threaded through every
456
  # aggregation above).
 
515
  ),
516
  )
517
 
518
+ st.pills(
519
  "Methods to display in charts",
520
  options=available_methods,
521
+ selection_mode="multi",
522
  key=k_methods,
523
+ help="Defaults to the top 5 methods by current rank — click to add or remove.",
524
  )
525
 
526
  selected_methods = st.session_state.get(k_methods, available_methods)
 
530
  st.divider()
531
 
532
  # --- Charts (rendered above the leaderboard table, below the controls) ---
533
+ # Failure panel — shown first so failures are visible before the budget/
534
+ # iterations charts below. Rendered regardless of whether the scored table
535
+ # has rows (own emptiness check on failure_source).
536
+ if show_failure_panel:
537
+ failure_source = filtered[filtered[target_str_col] == selected_target]
538
+ failure_source = failure_source[failure_source["abbreviation"].isin(selected_methods)]
539
+ if not failure_source.empty:
540
+ failure_df = failure_source.groupby(
541
+ ["abbreviation", "ensemble_size"], as_index=False
542
+ ).agg(mean_failure_rate=("failure_rate", "mean"))
543
+ failure_df = failure_df.sort_values("ensemble_size")
544
+
545
+ st.subheader(f"Failure Rate of Hitting Target {selected_target}")
546
+ ens_ticks_fail = sorted(failure_df["ensemble_size"].unique().tolist())
547
+ failure_chart = (
548
+ alt.Chart(failure_df)
549
+ .mark_bar()
550
+ .encode(
551
+ x=alt.X(
552
+ "ensemble_size:O",
553
+ title="Ensemble Size",
554
+ sort=[str(e) for e in ens_ticks_fail],
555
+ axis=alt.Axis(labelAngle=0),
556
+ ),
557
+ xOffset=alt.XOffset("abbreviation:N"),
558
+ y=alt.Y(
559
+ "mean_failure_rate:Q",
560
+ title="Failure Rate (%)",
561
+ scale=alt.Scale(domain=[0, 100]),
562
+ ),
563
+ color=_method_color(failure_df["abbreviation"].unique().tolist()),
564
+ tooltip=[
565
+ alt.Tooltip("abbreviation:N", title="Method"),
566
+ alt.Tooltip("ensemble_size:O", title="Ensemble Size"),
567
+ alt.Tooltip("mean_failure_rate:Q", format=".1f", title="Failure Rate (%)"),
568
+ ],
569
+ )
570
+ )
571
+ ceiling_line = (
572
+ alt.Chart(alt.Data(values=[{}]))
573
+ .mark_rule(color="grey", strokeDash=[4, 4])
574
+ .encode(y=alt.datum(100))
575
+ )
576
+ st.altair_chart(failure_chart + ceiling_line, use_container_width=True)
577
+ st.divider()
578
+
579
  if not leaderboard_df.empty:
580
  st.subheader("Mean Forward Model Runs vs Ensemble Size")
581
  chart_source = filtered[filtered[target_str_col] == selected_target]
 
667
 
668
  if chart_layers:
669
  st.altair_chart(alt.layer(*chart_layers), use_container_width=True)
670
+ st.caption(
671
+ "Note: plotted means here and in the charts below are averaged over "
672
+ "successful seeds only — runs that never reached the target are excluded. "
673
+ "An ✕ at zero marks ensemble sizes where every seed failed."
674
+ + (
675
+ " Failure frequency is shown in the failure-rate chart further down."
676
+ if show_failure_panel
677
+ else ""
678
+ )
679
+ )
680
+
681
+ # Mean-iterations section for the Optimization board: n_iterations =
682
+ # metric / ensemble_size, since every registered method's forward-model
683
+ # call count scales as ensemble_size (or restart count) per outer-loop
684
+ # iteration — see method_registry.py taxonomy. The UQ board instead
685
+ # renders its own iterations chart below from real k_iter data via
686
+ # budget_store, so this section only applies when none is supplied.
687
+ if budget_store is None and not all_ens_combos.empty:
688
+ iters_chart_df = chart_df.assign(
689
+ mean_iterations=lambda d: d["mean_forward_runs"] / d["ensemble_size"]
690
+ )
691
+ iters_fail_df = fail_df.assign(mean_iterations=0.0)
692
+
693
+ st.subheader("Mean Iterations to Reach Target vs Ensemble Size")
694
+ opt_iters_layers = []
695
+ if not iters_chart_df.empty:
696
+ opt_iters_layers.append(
697
+ alt.Chart(iters_chart_df)
698
+ .mark_line(point=True)
699
+ .encode(
700
+ x=alt.X(
701
+ "ensemble_size:Q",
702
+ title="Ensemble Size",
703
+ axis=alt.Axis(values=ens_ticks, format="d"),
704
+ ),
705
+ y=alt.Y("mean_iterations:Q", title="Mean Iterations"),
706
+ color=main_color,
707
+ tooltip=["abbreviation", "ensemble_size", alt.Tooltip("mean_iterations:Q", format=".2f")],
708
+ )
709
+ )
710
+ if not iters_fail_df.empty:
711
+ y_iters_opt = (
712
+ alt.Y("mean_iterations:Q", title="Mean Iterations", scale=alt.Scale(domain=[0, 1]))
713
+ if all_failed
714
+ else alt.Y("mean_iterations:Q", title="Mean Iterations")
715
+ )
716
+ opt_iters_layers.append(
717
+ alt.Chart(iters_fail_df)
718
+ .mark_point(shape="cross", angle=45, size=200, filled=True, opacity=1.0)
719
+ .encode(
720
+ x=alt.X(
721
+ "ensemble_size:Q",
722
+ title="Ensemble Size",
723
+ axis=alt.Axis(values=ens_ticks, format="d"),
724
+ ),
725
+ y=y_iters_opt,
726
+ color=main_color,
727
+ tooltip=[
728
+ alt.Tooltip("abbreviation:N", title="Method"),
729
+ alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
730
+ alt.Tooltip("mean_iterations:Q", title="Value (all failed)"),
731
+ ],
732
+ )
733
+ )
734
+ if opt_iters_layers:
735
+ st.altair_chart(alt.layer(*opt_iters_layers), use_container_width=True)
736
 
737
  # Mean-iterations-for-coverage section (UQ only, when budget_store provided)
738
  if budget_store is not None and not budget_store.empty:
 
808
  if iters_layers:
809
  st.altair_chart(alt.layer(*iters_layers), use_container_width=True)
810
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
811
  st.divider()
812
 
813
  # --- Leaderboard table (rendered below the charts) ---
src/common/method_registry.py CHANGED
@@ -134,6 +134,15 @@ KNOWN_METHODS = {
134
  "emulator_use": "none",
135
  "aliases": ["lm", "levenberg_marquardt", "levenberg-marquardt", "gradient_descent"],
136
  },
 
 
 
 
 
 
 
 
 
137
  }
138
 
139
 
@@ -142,6 +151,7 @@ KNOWN_METHODS = {
142
  _METHOD_PALETTE = [
143
  "#4c78a8", "#f58518", "#e45756", "#72b7b2", "#54a24b",
144
  "#eeca3b", "#b279a2", "#ff9da6", "#9d755d", "#bab0ac",
 
145
  ]
146
 
147
  # Stable abbreviation → hex color mapping. Import this wherever Altair charts are built
 
134
  "emulator_use": "none",
135
  "aliases": ["lm", "levenberg_marquardt", "levenberg-marquardt", "gradient_descent"],
136
  },
137
+ "gboed": {
138
+ "abbreviation": "GBOED",
139
+ "Method": "Goal-oriented Bayesian Optimal Experimental Design",
140
+ "parallelism": "parallel-interacting",
141
+ "update_type": "general",
142
+ "method_goal": "uq",
143
+ "emulator_use": "within-optimize",
144
+ "aliases": ["gboed", "boed"],
145
+ },
146
  }
147
 
148
 
 
151
  _METHOD_PALETTE = [
152
  "#4c78a8", "#f58518", "#e45756", "#72b7b2", "#54a24b",
153
  "#eeca3b", "#b279a2", "#ff9da6", "#9d755d", "#bab0ac",
154
+ "#5254a3", "#8ca252",
155
  ]
156
 
157
  # Stable abbreviation → hex color mapping. Import this wherever Altair charts are built
src/data_store.py CHANGED
@@ -93,6 +93,7 @@ UQ_BUDGET_FILES: dict[str, list[tuple[str, str]]] = {
93
  ("ces-eki-const", "ces-eki-const_results/ces-eki-const_l63_ensemble_results_2026-07-09_minimal.nc"),
94
  ("ces-iekf-const", "ces-iekf-const_results/ces-iekf-const_l63_ensemble_results_2026-07-17_minimal.nc"),
95
  ("hm", "history-matching-uq_results/history-matching_l63_ensemble_results_2026-07-20_minimal.nc"),
 
96
  ],
97
  "L96": [
98
  ("ces-eki-dmc", "ces-eki-dmc_results/ces-eki-dmc_l96_ensemble_results_2026-06-15_minimal.nc"),
@@ -100,6 +101,7 @@ UQ_BUDGET_FILES: dict[str, list[tuple[str, str]]] = {
100
  ("ces-eki-const", "ces-eki-const_results/ces-eki-const_l96_ensemble_results_2026-07-09_minimal.nc"),
101
  ("ces-iekf-const", "ces-iekf-const_results/ces-iekf-const_l96_ensemble_results_2026-07-17_minimal.nc"),
102
  ("hm", "history-matching-uq_results/history-matching_l96_const-force_2026-07-20_minimal.nc"),
 
103
  ],
104
  "L96_NN_FORCING": [
105
  ("ces-eki-dmc", "ces-eki-dmc_results/ces-eki-dmc_l96_nn_forcing_ensemble_results_2026-06-15_minimal.nc"),
@@ -114,6 +116,7 @@ UQ_BUDGET_FILES: dict[str, list[tuple[str, str]]] = {
114
  ("ces-eki-const", "ces-eki-const_results/ces-eki-const_l96_spatial_forcing_ensemble_results_2026-07-09_minimal.nc"),
115
  ("ces-iekf-const", "ces-iekf-const_results/ces-iekf-const_l96_spatial_forcing_ensemble_results_2026-07-17_minimal.nc"),
116
  ("hm", "history-matching-uq_results/history-matching_l96_vec-force_2026-07-22_minimal.nc"),
 
117
  ],
118
  }
119
 
 
93
  ("ces-eki-const", "ces-eki-const_results/ces-eki-const_l63_ensemble_results_2026-07-09_minimal.nc"),
94
  ("ces-iekf-const", "ces-iekf-const_results/ces-iekf-const_l63_ensemble_results_2026-07-17_minimal.nc"),
95
  ("hm", "history-matching-uq_results/history-matching_l63_ensemble_results_2026-07-20_minimal.nc"),
96
+ ("gboed", "gboed-uq_results/boed_l63_ensemble_results_2026-07-30_minimal.nc"),
97
  ],
98
  "L96": [
99
  ("ces-eki-dmc", "ces-eki-dmc_results/ces-eki-dmc_l96_ensemble_results_2026-06-15_minimal.nc"),
 
101
  ("ces-eki-const", "ces-eki-const_results/ces-eki-const_l96_ensemble_results_2026-07-09_minimal.nc"),
102
  ("ces-iekf-const", "ces-iekf-const_results/ces-iekf-const_l96_ensemble_results_2026-07-17_minimal.nc"),
103
  ("hm", "history-matching-uq_results/history-matching_l96_const-force_2026-07-20_minimal.nc"),
104
+ ("gboed", "gboed-uq_results/boed_l96_const-force_2026-07-30_minimal.nc"),
105
  ],
106
  "L96_NN_FORCING": [
107
  ("ces-eki-dmc", "ces-eki-dmc_results/ces-eki-dmc_l96_nn_forcing_ensemble_results_2026-06-15_minimal.nc"),
 
116
  ("ces-eki-const", "ces-eki-const_results/ces-eki-const_l96_spatial_forcing_ensemble_results_2026-07-09_minimal.nc"),
117
  ("ces-iekf-const", "ces-iekf-const_results/ces-iekf-const_l96_spatial_forcing_ensemble_results_2026-07-17_minimal.nc"),
118
  ("hm", "history-matching-uq_results/history-matching_l96_vec-force_2026-07-22_minimal.nc"),
119
+ ("gboed", "gboed-uq_results/boed_l96_vec-force_2026-08-07_minimal.nc"),
120
  ],
121
  }
122
 
src/pages/MethodDetails.py CHANGED
@@ -94,6 +94,11 @@ method_meta = {
94
  "url": "https://doi.org/10.1090/qam/10666",
95
  "summary": "Levenberg-Marquardt — damped least-squares algorithm that interpolates between gradient descent and Gauss-Newton steps for efficient nonlinear least-squares minimization.",
96
  },
 
 
 
 
 
97
  }
98
 
99
  # Selection UI (defaults to query param if valid)
 
94
  "url": "https://doi.org/10.1090/qam/10666",
95
  "summary": "Levenberg-Marquardt — damped least-squares algorithm that interpolates between gradient descent and Gauss-Newton steps for efficient nonlinear least-squares minimization.",
96
  },
97
+ "GBOED": {
98
+ "citation": "Holthuijzen et al. 2026",
99
+ "url": "https://arxiv.org/abs/2508.13071",
100
+ "summary": "Goal-oriented Bayesian Optimal Experimental Design — selects each batch of forward-model evaluations by maximizing an expected-information-gain criterion targeted at the quantity of interest, refitting the surrogate used by the criterion at every iteration.",
101
+ },
102
  }
103
 
104
  # Selection UI (defaults to query param if valid)
src/streamlit_app.py CHANGED
@@ -1,6 +1,17 @@
1
- import streamlit as st
2
  from pathlib import Path
3
 
 
 
 
 
 
 
 
 
 
 
 
4
  st.set_page_config(page_title="Calibration Benchmark", page_icon="🏠", layout="wide")
5
 
6
  # Sidebar navigation
@@ -88,38 +99,51 @@ st.markdown(
88
  "a surrogate model. See the **📘 Methods** page for citations and per-method "
89
  "performance charts."
90
  )
91
- st.markdown(
92
- """
93
- - **Parallelism** — how the search explores parameter space
94
- - **Serial** — `ADAM`, `LM` — a single point estimate advanced step by step.
95
- - **Parallel-independent** — `ABC`, `HM` — a population of candidates updated
96
- with no coupling between members (accepted samples / per-wave resampling).
97
- - **Parallel-interacting** — `TEKI`, `ETKI`, `IEKF`, `UKI`, `CES-EKI-DMC`, `CES-EKI-CONST`,
98
- `CES-IEKF-CONST` — an ensemble whose members are coupled through a shared update
99
- each iteration.
100
- - **Update type** — the mechanism driving each update step
101
- - **Gradient** — `ADAM`, `LM` — follow the loss gradient (or a Gauss-Newton
102
- approximation of it) directly.
103
- - **Kalman** — `TEKI`, `ETKI`, `IEKF`, `UKI`, `CES-EKI-DMC`, `CES-EKI-CONST`,
104
- `CES-IEKF-CONST` — a (possibly linearized or unscented) Kalman-style ensemble update.
105
- - **General** — `ABC`, `HM` — neither gradient- nor Kalman-based (rejection
106
- sampling, implausibility cuts).
107
- - **Method goal** — what the method is built to report
108
- - **Optimization** — `TEKI`, `ETKI`, `UKI`, `ADAM`, `LM` — a single best-fit
109
- parameter estimate.
110
- - **UQ** — `IEKF`, `ABC`, `HM`, `CES-EKI-DMC`, `CES-EKI-CONST`, `CES-IEKF-CONST` — the
111
- full posterior / parameter uncertainty. Can still be scored on the Optimization
112
- leaderboard, but tends to be less competitive there since it's not optimizing for
113
- speed-to-target.
114
- - **Emulator use** — when/whether a surrogate model of the forward model is used
115
- - **None** — `TEKI`, `ETKI`, `IEKF`, `UKI`, `ADAM`, `LM`, `ABC` — samples/evaluates
116
- the true forward model throughout.
117
- - **Within-optimize** — `HM` — refits a surrogate at each iteration (wave) of the
118
- search itself.
119
- - **After-optimize** — `CES-EKI-DMC`, `CES-EKI-CONST`, `CES-IEKF-CONST` — fits a
120
- surrogate (e.g. a GP) once, after calibration finishes, and samples the posterior
121
- through it.
122
- """
 
 
 
 
 
 
 
 
 
 
 
 
 
123
  )
124
  st.caption(
125
  "Note: Kalman methods are Bayesian in spirit too (they're approximate Gaussian "
 
1
+ import sys
2
  from pathlib import Path
3
 
4
+ import pandas as pd
5
+ import streamlit as st
6
+
7
+ try:
8
+ from common.method_registry import KNOWN_METHODS
9
+ from common.leaderboard import _UPDATE_TYPE_ORDER
10
+ except ModuleNotFoundError:
11
+ sys.path.append(str(Path(__file__).resolve().parent))
12
+ from common.method_registry import KNOWN_METHODS
13
+ from common.leaderboard import _UPDATE_TYPE_ORDER
14
+
15
  st.set_page_config(page_title="Calibration Benchmark", page_icon="🏠", layout="wide")
16
 
17
  # Sidebar navigation
 
99
  "a surrogate model. See the **📘 Methods** page for citations and per-method "
100
  "performance charts."
101
  )
102
+ _taxonomy_rows = [
103
+ {
104
+ "Algorithm": info["abbreviation"],
105
+ "Method Goal": info["method_goal"],
106
+ "Update Type": info["update_type"],
107
+ "Parallelism": info["parallelism"],
108
+ "Emulator Use": info["emulator_use"],
109
+ "_update_type_order": _UPDATE_TYPE_ORDER.get(info["update_type"], 99),
110
+ }
111
+ for info in KNOWN_METHODS.values()
112
+ ]
113
+ _taxonomy_df = (
114
+ pd.DataFrame(_taxonomy_rows)
115
+ .sort_values(["_update_type_order", "Algorithm"])
116
+ .drop(columns="_update_type_order")
117
+ )
118
+
119
+ _UPDATE_TYPE_ROW_COLOR = {
120
+ "kalman": "#d9f2e6", # pastel green
121
+ "gradient": "#dbe9fa", # pastel blue
122
+ "general": "#fde2c8", # pastel orange
123
+ }
124
+
125
+
126
+ def _style_taxonomy_rows(df: pd.DataFrame) -> pd.DataFrame:
127
+ result = pd.DataFrame("", index=df.index, columns=df.columns)
128
+ for row in df.index:
129
+ bg = _UPDATE_TYPE_ROW_COLOR.get(df.loc[row, "Update Type"], "")
130
+ if bg:
131
+ result.loc[row, :] = f"background-color: {bg}; color: #212529"
132
+ return result
133
+
134
+
135
+ st.dataframe(
136
+ _taxonomy_df.style.apply(_style_taxonomy_rows, axis=None),
137
+ hide_index=True,
138
+ use_container_width=True,
139
+ height=int((len(_taxonomy_df) + 1) * 35.2 + 3),
140
+ column_config={
141
+ "Algorithm": st.column_config.TextColumn("Algorithm"),
142
+ "Method Goal": st.column_config.TextColumn("Method Goal"),
143
+ "Update Type": st.column_config.TextColumn("Update Type"),
144
+ "Parallelism": st.column_config.TextColumn("Parallelism"),
145
+ "Emulator Use": st.column_config.TextColumn("Emulator Use"),
146
+ },
147
  )
148
  st.caption(
149
  "Note: Kalman methods are Bayesian in spirit too (they're approximate Gaussian "