odunbar commited on
Commit
dc2991f
Β·
1 Parent(s): c29359c

change methods taxonomy

Browse files
.cache/known_methods_snapshot.json CHANGED
@@ -3,7 +3,8 @@
3
  "teki": {
4
  "abbreviation": "TEKI",
5
  "Method": "Tikhonov Regularized Ensemble Kalman Inversion",
6
- "family": "Kalman",
 
7
  "aliases": [
8
  "teki"
9
  ]
@@ -11,7 +12,8 @@
11
  "etki": {
12
  "abbreviation": "ETKI",
13
  "Method": "Ensemble Transform Kalman Inversion",
14
- "family": "Kalman",
 
15
  "aliases": [
16
  "etki"
17
  ]
@@ -19,7 +21,8 @@
19
  "iekf": {
20
  "abbreviation": "IEKF",
21
  "Method": "Iterative Ensemble Kalman Filter",
22
- "family": "Kalman",
 
23
  "aliases": [
24
  "iekf",
25
  "gnsl",
@@ -29,7 +32,8 @@
29
  "uki": {
30
  "abbreviation": "UKI",
31
  "Method": "Unscented Kalman Inversion",
32
- "family": "Kalman",
 
33
  "aliases": [
34
  "uki"
35
  ]
@@ -37,7 +41,8 @@
37
  "abc": {
38
  "abbreviation": "ABC",
39
  "Method": "Approximate Bayesian Calibration",
40
- "family": "Bayesian",
 
41
  "aliases": [
42
  "abc"
43
  ]
@@ -45,7 +50,8 @@
45
  "hm": {
46
  "abbreviation": "HM",
47
  "Method": "History Matching",
48
- "family": "Bayesian",
 
49
  "aliases": [
50
  "hm"
51
  ]
@@ -53,7 +59,8 @@
53
  "ces-eki-dmc": {
54
  "abbreviation": "CES-EKI-DMC",
55
  "Method": "Calibrate Emulate Sample (EKI-DataMisfitController)",
56
- "family": "calibrate_then_emulate",
 
57
  "aliases": [
58
  "ces-eki-dmc"
59
  ]
@@ -61,6 +68,7 @@
61
  "adam": {
62
  "abbreviation": "ADAM",
63
  "Method": "Adaptive Moment Estimation",
 
64
  "family": "gradient",
65
  "aliases": [
66
  "adam"
@@ -69,6 +77,7 @@
69
  "lm": {
70
  "abbreviation": "LM",
71
  "Method": "Levenberg-Marquardt",
 
72
  "family": "gradient",
73
  "aliases": [
74
  "lm",
 
3
  "teki": {
4
  "abbreviation": "TEKI",
5
  "Method": "Tikhonov Regularized Ensemble Kalman Inversion",
6
+ "method_class": "optimizer",
7
+ "family": "kalman",
8
  "aliases": [
9
  "teki"
10
  ]
 
12
  "etki": {
13
  "abbreviation": "ETKI",
14
  "Method": "Ensemble Transform Kalman Inversion",
15
+ "method_class": "optimizer",
16
+ "family": "kalman",
17
  "aliases": [
18
  "etki"
19
  ]
 
21
  "iekf": {
22
  "abbreviation": "IEKF",
23
  "Method": "Iterative Ensemble Kalman Filter",
24
+ "method_class": "optimizer",
25
+ "family": "kalman",
26
  "aliases": [
27
  "iekf",
28
  "gnsl",
 
32
  "uki": {
33
  "abbreviation": "UKI",
34
  "Method": "Unscented Kalman Inversion",
35
+ "method_class": "optimizer",
36
+ "family": "kalman",
37
  "aliases": [
38
  "uki"
39
  ]
 
41
  "abc": {
42
  "abbreviation": "ABC",
43
  "Method": "Approximate Bayesian Calibration",
44
+ "method_class": "uq",
45
+ "family": "uq-non-emulator",
46
  "aliases": [
47
  "abc"
48
  ]
 
50
  "hm": {
51
  "abbreviation": "HM",
52
  "Method": "History Matching",
53
+ "method_class": "uq",
54
+ "family": "uq-emulate-within-optimize",
55
  "aliases": [
56
  "hm"
57
  ]
 
59
  "ces-eki-dmc": {
60
  "abbreviation": "CES-EKI-DMC",
61
  "Method": "Calibrate Emulate Sample (EKI-DataMisfitController)",
62
+ "method_class": "uq",
63
+ "family": "uq-emulate-after-optimize",
64
  "aliases": [
65
  "ces-eki-dmc"
66
  ]
 
68
  "adam": {
69
  "abbreviation": "ADAM",
70
  "Method": "Adaptive Moment Estimation",
71
+ "method_class": "optimizer",
72
  "family": "gradient",
73
  "aliases": [
74
  "adam"
 
77
  "lm": {
78
  "abbreviation": "LM",
79
  "Method": "Levenberg-Marquardt",
80
+ "method_class": "optimizer",
81
  "family": "gradient",
82
  "aliases": [
83
  "lm",
CLAUDE.md CHANGED
@@ -96,7 +96,7 @@ calibration_benchmark/
96
  | `algorithm_type` | Canonical method key (e.g. `teki`) |
97
  | `algorithm_alias` | Normalized raw name from NetCDF |
98
  | `abbreviation` | Display name (e.g. `TEKI`) |
99
- | `family` | `Kalman` Β· `Bayesian` Β· `calibrate_then_emulate` Β· `gradient` |
100
  | `rmse_target` | Target RMSE level |
101
  | `ensemble_size` | Ensemble / particle count |
102
  | `metric` | Mean forward-model runs over `random_seed` (lower = better; βˆ’1 = failed) |
@@ -149,25 +149,46 @@ Physical dimensions of each benchmark exported from `src/data_store.py` as
149
  Passed as `benchmark_dims=BENCHMARK_DIMS` to `render_leaderboard()` for column-header
150
  annotations in the suitability table. Update if a new benchmark is added.
151
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
152
  ## Method registry (`src/common/method_registry.py`)
153
 
154
- | Canonical | Abbreviation | Family | Aliases |
155
- |---|---|---|---|
156
- | `teki` | TEKI | Kalman | β€” |
157
- | `etki` | ETKI | Kalman | β€” |
158
- | `iekf` | IEKF | Kalman | gnsl, gnki |
159
- | `uki` | UKI | Kalman | β€” |
160
- | `abc` | ABC | Bayesian | β€” |
161
- | `hm` | HM | Bayesian | β€” |
162
- | `ces-eki-dmc` | CES-EKI-DMC | calibrate_then_emulate | β€” |
163
- | `adam` | ADAM | gradient | β€” |
164
- | `lm` | LM | gradient | gradient_descent |
165
 
166
  ## How to extend
167
 
168
  ### Add a method
169
  1. Add entry to `KNOWN_METHODS` in `src/common/method_registry.py`
170
- (keys: `abbreviation`, `Method`, `family`, `aliases`).
 
171
  2. Add detail-page metadata to the inline `method_meta` dict in `src/pages/MethodDetails.py`
172
  keyed by **abbreviation** (e.g. `"TEKI"`). ⚠️ Metadata is split across both files.
173
  3. Place result NetCDF(s) in `data/` and add paths to `DATASET_FILES` in `src/data_store.py`.
 
96
  | `algorithm_type` | Canonical method key (e.g. `teki`) |
97
  | `algorithm_alias` | Normalized raw name from NetCDF |
98
  | `abbreviation` | Display name (e.g. `TEKI`) |
99
+ | `family` | Structural sub-type β€” see **Method taxonomy** below |
100
  | `rmse_target` | Target RMSE level |
101
  | `ensemble_size` | Ensemble / particle count |
102
  | `metric` | Mean forward-model runs over `random_seed` (lower = better; βˆ’1 = failed) |
 
149
  Passed as `benchmark_dims=BENCHMARK_DIMS` to `render_leaderboard()` for column-header
150
  annotations in the suitability table. Update if a new benchmark is added.
151
 
152
+ ## Method taxonomy
153
+
154
+ Two orthogonal fields on every `KNOWN_METHODS` entry:
155
+
156
+ - **`method_class`** β€” `optimizer` (drives toward a single best-fit estimate) or `uq`
157
+ (characterizes the full posterior/uncertainty). UQ methods *can* still be scored on
158
+ the Optimization leaderboard β€” they're just usually less competitive there.
159
+ - **`family`** β€” structural mechanism, scoped within `method_class`:
160
+ - `optimizer` β†’ `gradient` Β· `kalman` Β· `ensemble-other` (non-Kalman ensemble;
161
+ unoccupied today β€” reserved for e.g. CMA-ES/particle-swarm methods)
162
+ - `uq` β†’ `uq-non-emulator` (samples the true forward model directly) Β·
163
+ `uq-emulate-after-optimize` (fits a surrogate once, after calibration) Β·
164
+ `uq-emulate-within-optimize` (refits a surrogate at each search iteration)
165
+
166
+ `family` deliberately avoids a "Bayesian" label β€” Kalman methods are approximate
167
+ Bayesian updates too, so the split is about mechanism (gradient vs. ensemble vs.
168
+ emulator timing), not philosophy. `_FAMILY_ORDER` in `src/common/leaderboard.py`
169
+ controls table/suitability sort order and must stay in sync with these `family` values.
170
+ A quick taxonomy tree is rendered on the home page (`src/streamlit_app.py`).
171
+
172
  ## Method registry (`src/common/method_registry.py`)
173
 
174
+ | Canonical | Abbreviation | method_class | family | Aliases |
175
+ |---|---|---|---|---|
176
+ | `teki` | TEKI | optimizer | kalman | β€” |
177
+ | `etki` | ETKI | optimizer | kalman | β€” |
178
+ | `iekf` | IEKF | optimizer | kalman | gnsl, gnki |
179
+ | `uki` | UKI | optimizer | kalman | β€” |
180
+ | `abc` | ABC | uq | uq-non-emulator | β€” |
181
+ | `hm` | HM | uq | uq-emulate-within-optimize | β€” |
182
+ | `ces-eki-dmc` | CES-EKI-DMC | uq | uq-emulate-after-optimize | β€” |
183
+ | `adam` | ADAM | optimizer | gradient | β€” |
184
+ | `lm` | LM | optimizer | gradient | gradient_descent |
185
 
186
  ## How to extend
187
 
188
  ### Add a method
189
  1. Add entry to `KNOWN_METHODS` in `src/common/method_registry.py`
190
+ (keys: `abbreviation`, `Method`, `method_class`, `family`, `aliases` β€” see
191
+ **Method taxonomy** above for valid `method_class`/`family` values).
192
  2. Add detail-page metadata to the inline `method_meta` dict in `src/pages/MethodDetails.py`
193
  keyed by **abbreviation** (e.g. `"TEKI"`). ⚠️ Metadata is split across both files.
194
  3. Place result NetCDF(s) in `data/` and add paths to `DATASET_FILES` in `src/data_store.py`.
src/common/leaderboard.py CHANGED
@@ -51,11 +51,16 @@ def _method_color(present_abbrevs: list[str]) -> alt.Color:
51
 
52
  # Family display order in the leaderboard table (lower = earlier).
53
  # Unknown families fall back to 99 and appear at the end.
 
 
 
54
  _FAMILY_ORDER: dict[str, int] = {
55
- "Kalman": 0,
56
- "Bayesian": 1,
57
- "gradient": 2,
58
- "calibrate_then_emulate": 3,
 
 
59
  }
60
 
61
 
 
51
 
52
  # Family display order in the leaderboard table (lower = earlier).
53
  # Unknown families fall back to 99 and appear at the end.
54
+ # Optimizer families first (gradient, kalman, ensemble-other), then UQ families
55
+ # ordered non-emulator -> emulate-after -> emulate-within. See method_registry.py
56
+ # for the full taxonomy (method_class + family).
57
  _FAMILY_ORDER: dict[str, int] = {
58
+ "gradient": 0,
59
+ "kalman": 1,
60
+ "ensemble-other": 2,
61
+ "uq-non-emulator": 3,
62
+ "uq-emulate-after-optimize": 4,
63
+ "uq-emulate-within-optimize": 5,
64
  }
65
 
66
 
src/common/method_registry.py CHANGED
@@ -4,58 +4,84 @@ import json
4
  from pathlib import Path
5
 
6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  KNOWN_METHODS = {
8
  "teki": {
9
  "abbreviation": "TEKI",
10
  "Method": "Tikhonov Regularized Ensemble Kalman Inversion",
11
- "family": "Kalman",
 
12
  "aliases": ["teki"],
13
  },
14
  "etki": {
15
  "abbreviation": "ETKI",
16
  "Method": "Ensemble Transform Kalman Inversion",
17
- "family": "Kalman",
 
18
  "aliases": ["etki"],
19
  },
20
  "iekf": {
21
  "abbreviation": "IEKF",
22
  "Method": "Iterative Ensemble Kalman Filter",
23
- "family": "Kalman",
 
24
  "aliases": ["iekf", "gnsl", "gnki"],
25
  },
26
  "uki": {
27
  "abbreviation": "UKI",
28
  "Method": "Unscented Kalman Inversion",
29
- "family": "Kalman",
 
30
  "aliases": ["uki"],
31
  },
32
  "abc": {
33
  "abbreviation": "ABC",
34
  "Method": "Approximate Bayesian Calibration",
35
- "family": "Bayesian",
 
36
  "aliases": ["abc"],
37
  },
38
  "hm": {
39
  "abbreviation": "HM",
40
  "Method": "History Matching",
41
- "family": "Bayesian",
 
42
  "aliases": ["hm"],
43
  },
44
  "ces-eki-dmc": {
45
  "abbreviation": "CES-EKI-DMC",
46
  "Method": "Calibrate Emulate Sample (EKI-DataMisfitController)",
47
- "family": "calibrate_then_emulate",
 
48
  "aliases": ["ces-eki-dmc"]
49
  },
50
  "adam": {
51
  "abbreviation": "ADAM",
52
  "Method": "Adaptive Moment Estimation",
 
53
  "family": "gradient",
54
  "aliases": ["adam"],
55
  },
56
  "lm": {
57
  "abbreviation": "LM",
58
  "Method": "Levenberg-Marquardt",
 
59
  "family": "gradient",
60
  "aliases": ["lm", "levenberg_marquardt", "levenberg-marquardt", "gradient_descent"],
61
  },
 
4
  from pathlib import Path
5
 
6
 
7
+ # Taxonomy
8
+ # --------
9
+ # ``method_class`` β€” top-level split, independent of which leaderboard a method's
10
+ # data happens to be registered on:
11
+ # "optimizer" β€” drives toward a single best-fit parameter estimate.
12
+ # "uq" β€” characterizes the full posterior / parameter uncertainty.
13
+ # UQ methods can still be scored on the Optimization leaderboard
14
+ # (they usually just aren't as competitive there).
15
+ #
16
+ # ``family`` β€” structural sub-type within a method_class:
17
+ # optimizer: "gradient" Β· "kalman" Β· "ensemble-other" (non-Kalman ensemble; no
18
+ # method occupies this slot yet, reserved for e.g. CMA-ES/particle swarm)
19
+ # uq: "uq-non-emulator" (posterior sampling, no surrogate model) Β·
20
+ # "uq-emulate-after-optimize" (build a surrogate once optimization/
21
+ # calibration finishes, e.g. CES) Β·
22
+ # "uq-emulate-within-optimize" (refit/refine a surrogate at each
23
+ # iteration of the search, e.g. History Matching waves)
24
  KNOWN_METHODS = {
25
  "teki": {
26
  "abbreviation": "TEKI",
27
  "Method": "Tikhonov Regularized Ensemble Kalman Inversion",
28
+ "method_class": "optimizer",
29
+ "family": "kalman",
30
  "aliases": ["teki"],
31
  },
32
  "etki": {
33
  "abbreviation": "ETKI",
34
  "Method": "Ensemble Transform Kalman Inversion",
35
+ "method_class": "optimizer",
36
+ "family": "kalman",
37
  "aliases": ["etki"],
38
  },
39
  "iekf": {
40
  "abbreviation": "IEKF",
41
  "Method": "Iterative Ensemble Kalman Filter",
42
+ "method_class": "optimizer",
43
+ "family": "kalman",
44
  "aliases": ["iekf", "gnsl", "gnki"],
45
  },
46
  "uki": {
47
  "abbreviation": "UKI",
48
  "Method": "Unscented Kalman Inversion",
49
+ "method_class": "optimizer",
50
+ "family": "kalman",
51
  "aliases": ["uki"],
52
  },
53
  "abc": {
54
  "abbreviation": "ABC",
55
  "Method": "Approximate Bayesian Calibration",
56
+ "method_class": "uq",
57
+ "family": "uq-non-emulator",
58
  "aliases": ["abc"],
59
  },
60
  "hm": {
61
  "abbreviation": "HM",
62
  "Method": "History Matching",
63
+ "method_class": "uq",
64
+ "family": "uq-emulate-within-optimize",
65
  "aliases": ["hm"],
66
  },
67
  "ces-eki-dmc": {
68
  "abbreviation": "CES-EKI-DMC",
69
  "Method": "Calibrate Emulate Sample (EKI-DataMisfitController)",
70
+ "method_class": "uq",
71
+ "family": "uq-emulate-after-optimize",
72
  "aliases": ["ces-eki-dmc"]
73
  },
74
  "adam": {
75
  "abbreviation": "ADAM",
76
  "Method": "Adaptive Moment Estimation",
77
+ "method_class": "optimizer",
78
  "family": "gradient",
79
  "aliases": ["adam"],
80
  },
81
  "lm": {
82
  "abbreviation": "LM",
83
  "Method": "Levenberg-Marquardt",
84
+ "method_class": "optimizer",
85
  "family": "gradient",
86
  "aliases": ["lm", "levenberg_marquardt", "levenberg-marquardt", "gradient_descent"],
87
  },
src/data_store.py CHANGED
@@ -232,7 +232,7 @@ def _load_store(dataset_files: dict[str, list[str]], target_col: str, *, drop_na
232
  merged["family"] = merged["algorithm_type"].map(lambda method: get_method_meta(method).get("family"))
233
  merged["abbreviation"] = merged["abbreviation"].fillna(merged["algorithm_type"].str.upper())
234
  merged["Method"] = merged["Method"].fillna(merged["abbreviation"])
235
- merged["family"] = merged["family"].fillna("Kalman")
236
 
237
  try:
238
  dump_method_registry_snapshot(project_root=project_root, observed_methods=set(merged["algorithm_type"].unique()))
@@ -395,6 +395,6 @@ def load_uq_budget_store() -> pd.DataFrame:
395
  return get_method_meta(canonicalize_method_name(normalize_method_name(m))).get(key, fallback)
396
 
397
  df["abbreviation"] = df["algorithm_type"].map(lambda m: _meta(m, "abbreviation", m.upper()))
398
- df["family"] = df["algorithm_type"].map(lambda m: _meta(m, "family", "Kalman"))
399
  df["Method"] = df["algorithm_type"].map(lambda m: _meta(m, "Method", m.upper()))
400
  return df
 
232
  merged["family"] = merged["algorithm_type"].map(lambda method: get_method_meta(method).get("family"))
233
  merged["abbreviation"] = merged["abbreviation"].fillna(merged["algorithm_type"].str.upper())
234
  merged["Method"] = merged["Method"].fillna(merged["abbreviation"])
235
+ merged["family"] = merged["family"].fillna("unmapped")
236
 
237
  try:
238
  dump_method_registry_snapshot(project_root=project_root, observed_methods=set(merged["algorithm_type"].unique()))
 
395
  return get_method_meta(canonicalize_method_name(normalize_method_name(m))).get(key, fallback)
396
 
397
  df["abbreviation"] = df["algorithm_type"].map(lambda m: _meta(m, "abbreviation", m.upper()))
398
+ df["family"] = df["algorithm_type"].map(lambda m: _meta(m, "family", "unmapped"))
399
  df["Method"] = df["algorithm_type"].map(lambda m: _meta(m, "Method", m.upper()))
400
  return df
src/streamlit_app.py CHANGED
@@ -81,18 +81,37 @@ st.image(
81
  width=1200,
82
  )
83
 
84
- st.subheader("Method families")
85
  st.markdown(
86
- "Methods are grouped into three families. See the **πŸ“˜ Methods** page for "
87
- "citations and per-method performance charts."
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  )
89
- families = {
90
- "Kalman": "Ensemble Kalman variants (TEKI, ETKI, IEKF, UKI) β€” update an ensemble of parameter guesses via a linearized observation operator.",
91
- "Bayesian": "Sampling-based approaches (ABC, HM) β€” explore parameter space without requiring gradient information.",
92
- "Calibrate-then-emulate": "Two-stage pipelines (CES-EKI-DMC) β€” use an initial calibration phase to build a cheap emulator, then sample the posterior via MCMC.",
93
- }
94
- for family, desc in families.items():
95
- st.markdown(f"- **{family}** β€” {desc}")
96
 
97
  st.subheader("Key metric")
98
  st.markdown(
 
81
  width=1200,
82
  )
83
 
84
+ st.subheader("Method taxonomy")
85
  st.markdown(
86
+ "Methods split first by what they're built to do, then by how they get there. "
87
+ "See the **πŸ“˜ Methods** page for citations and per-method performance charts."
88
+ )
89
+ st.markdown(
90
+ """
91
+ - **Optimizer methods** β€” drive toward a single best-fit parameter estimate
92
+ - **Gradient-based** β€” `ADAM`, `LM` β€” follow the loss gradient (or a Gauss-Newton
93
+ approximation of it) directly.
94
+ - **Ensemble-based** β€” update a population of parameter guesses instead of one
95
+ - **Kalman** β€” `TEKI`, `ETKI`, `IEKF`, `UKI` β€” update the ensemble via a
96
+ (possibly linearized or unscented) Kalman step.
97
+ - **Non-Kalman** β€” *(none registered yet β€” e.g. genetic/CMA-ES/particle-swarm
98
+ methods would land here)*
99
+ - **UQ methods** β€” characterize the full posterior / parameter uncertainty, not just
100
+ a point estimate. Can also be scored on the Optimization leaderboard, but tend to be
101
+ less competitive there since they're not optimizing for speed-to-target.
102
+ - **Non-emulator** β€” `ABC` β€” sample the posterior directly against the true
103
+ forward model, no surrogate involved.
104
+ - **Emulate-after-optimize** β€” `CES-EKI-DMC` β€” calibrate first, then fit a
105
+ surrogate (e.g. a GP) once and sample the posterior through it.
106
+ - **Emulate-within-optimize** β€” `HM` β€” refit/refine a surrogate at each
107
+ iteration (wave) of the search itself.
108
+ """
109
+ )
110
+ st.caption(
111
+ "Note: Kalman methods are Bayesian in spirit too (they're approximate Gaussian "
112
+ "posterior updates) β€” the split above is about mechanism (gradient vs. ensemble vs. "
113
+ "emulator timing), not whether a method is 'Bayesian'."
114
  )
 
 
 
 
 
 
 
115
 
116
  st.subheader("Key metric")
117
  st.markdown(