# Which recipe, and when to search The catalog holds only recipes with rows in the saved result files (`results/`). Every number on this page comes from those files through `benchmarks/catalog_summary.py` (output: `benchmarks/results/catalog_summary.json`), from this package's own timing benchmark (`benchmarks/results/speed_engines.csv`, see [SPEED.md](SPEED.md)), or from the cost–accuracy figure data (`docs/figures/cost_accuracy_plotted_values.csv`); each statement says which. Paper averages below follow the paper's rule: mean over the 5 folds and 3 seeds (42, 43, 44) per view, then over the nine primary dataset views (the two HARMES wrist streams form one view; the nine-channel mHealth and PAMAP2 streams are not in this average). This rule reproduces the reported means exactly (e.g. WISP-CIS macro-F1 66.1391). They are test-split results of the paper protocol and are listed for orientation only: none of the advice on this page was chosen by looking at them (see "How to choose" below). ## How to choose 1. **Start with `"cis"` (WISP-CIS, the default).** It is the recipe that validation most often picks: WISP-Select5 chose CIS in 106 of its 180 validation selections (SS 44, RC 29, GIS 1; `models/model_inventory.csv` of the Hugging Face repository), and among the 360 final recipes of the budget-329 searches, each the validation winner of its search, the fixed WISP-CIS configuration is the most frequent one (53 of 360; `results/search_selections.csv`). It needs no search. 2. **Let validation choose among the fixed families with `WISPSelect(menu=5)`** when you have a validation split of separate participants. It fits five candidates on training data and refits the winner on train+validation. 3. **Try the search's best recipe of a similar data stream** (`wisp.recipes.best(...)`, table below), e.g. for similar sensors, window length and activity set. 4. **Search last, and try WISP-Funnel first** (`WISPSearch("funnel")`, [below](#wisp-funnel-a-fast-search)). One funnel run gives both a Random and an Evolution model for about 1/47 of the candidate-fitting time of one paper WISP-Random search, within one macro-F1 point of the full searches on the benchmark. The full searches (`WISPSearch("random" | "evolution")`) are expensive: on the ten primary streams the median cumulative candidate fitting time of one paper search (one dataset, fold and seed, 329 candidates) was 47.5 h for WISP-Random and 61.3 h for WISP-Evolution (`results/search_selections.csv`, budget 329, 150 searches each). The final model of a search is cheap; the cost is in the selection. Pass `groups` (and `time_index`) at fit and predict time when windows come in sequences: the default `decoder="adaptive"` then decides on validation data whether the HMM decoder helps (see [API.md](API.md#the-adaptive-hmm-switch-decoderadaptive)). In the paper, each recipe carried its own HMM flag; about four in five of the 360 final search recipes kept the HMM (Random 146/180, Evolution 144/180). The paper numbers on this page use each recipe's own decoder: reproduce them with `decoder="recipe"`. ## Fixed families | Recipe | Paper name | Macro-F1 | Balanced acc. | Accuracy | Worst-class F1 | |---|---|---:|---:|---:|---:| | `"ss"` | WISP-SS | 58.54 | 59.04 | 73.36 | 21.70 | | `"rc"` | WISP-RC | 63.44 | 63.59 | 71.46 | 22.87 | | `"so"` | WISP-SO | 53.59 | 54.19 | 69.31 | 16.75 | | `"gis"` | WISP-GIS | 55.16 | 55.95 | 70.66 | 19.49 | | `"cis"` | WISP-CIS | 66.14 | 66.04 | 74.07 | 24.78 | | `"cse"` | WISP-CSE | 58.74 | 59.10 | 72.86 | 21.22 | | `"ese"` | WISP-ESE | 57.38 | 57.71 | 71.73 | 19.78 | | `WISPSelect(5)` | WISP-Select5 | 67.10 | 66.94 | 75.21 | 24.93 | | `WISPSelect(7)` | WISP-Select7 | 66.69 | 66.65 | 75.01 | 24.65 | | `WISPSearch("random")` | WISP-Random | 70.20 | 70.88 | 77.28 | 28.25 | | `WISPSearch("evolution")` | WISP-Evolution | 69.59 | 70.17 | 76.84 | 27.96 | Source: `results/run_results.csv`, paper average (percent). All fixed families use the HMM decoder (`decoder="recipe"` reproduces these numbers). ## CIS block subsets `"cis:R"`, `"cis:I"`, `"cis:S"`, `"cis:RI"`, `"cis:RS"`, `"cis:IS"` (and `"cis:RIS"`, which is `"cis"`) keep only the named blocks of WISP-CIS: R random convolution, I interval summaries, S within-window state and spectral features. Each was run with and without the HMM decoder; switch with `decoder="hmm"` / `decoder="none"`. | Blocks | Macro-F1, HMM | Macro-F1, direct | |---|---:|---:| | R | 63.44 | 57.28 | | I | 43.89 | 38.63 | | S | 59.39 | 55.30 | | RI | 63.59 | 57.65 | | RS | 66.93 | 60.87 | | IS | 61.42 | 56.25 | | RIS (= `"cis"`) | 66.14 | 61.12 | Source: `results/cis_and_budget_runs.csv` (`kind == "recipe"`), paper average. These are a controlled block study, not a recommendation: they are test results, so do not pick a subset from this table for a new study; choose it on your own validation data (e.g. with `GridSearchCV` over `recipe`). ## Search best recipes Each search returns its validation winner. The best recipe of a data stream is the final recipe selected most often in its 30 searches (5 folds x 3 seeds x WISP-Random/WISP-Evolution), ties broken by the number of folds covered. Only validation decides; no test score is read. `scripts/build_search_best.py` computes the table from `results/search_selections.csv` and writes `src/wisp/_data/search_best.json`. | `wisp.recipes.best(...)` | Recipe | Selected | Folds covered | Same as | |---|---|---:|---:|---| | `"adl"` | RC, 256 kernels, Ridge, HMM | 9/30 | 3/5 | — (one of the search's initial templates) | | `"capture24"` | CIS, 128 kernels, 24 intervals, Ridge, HMM | 8/30 | 4/5 | — | | `"domino"` | CIS, 384 kernels, 24 intervals, Ridge, HMM | 11/30 | 4/5 | `"cis"` | | `"gotov"` | same as DOMINO | 20/30 | 5/5 | `"cis"` | | `"iuwds"` | same as DOMINO | 12/30 | 5/5 | `"cis"` | | `"harmes_left"` | CIS, 384 kernels, 24 intervals, Ridge, no HMM | 8/30 | 5/5 | `"cis"` with `decoder="none"` | | `"harmes_right"` | same as HARMES left | 7/30 | 4/5 | `"cis"` with `decoder="none"` | | `"handy"` | SO, HMM | 9/30 | 3/5 | `"so"` | | `"wisdm"` | GIS, ExtraTrees, 32 shapelets, HMM | 15/30 | 3/5 | `"gis"` | | `"paal_adl"` | SS, random forest, segments (5, 10, 20), 8 states, HMM | 8/30 | 4/5 | — (one of the search's initial templates) | | `"pamap2"` | RC, 256 kernels, Ridge, HMM | 6/30 | 3/5 | — (one of the search's initial templates) | | `"mhealth"` | SO, HMM | 6/30 | **1/5** | `"so"` | Notes: * Eight of the twelve are identical to a fixed recipe or to `"cis"` without the HMM, so they have results on every fold and seed. The other four (ADL, CAPTURE-24, PAAL-ADL, PAMAP2) have results only in the searches that selected them. * The mHealth best recipe comes from a single fold; its support is weak. * Across all 360 searches the most frequent final recipe is the fixed WISP-CIS (53 times), one reason it is the package default. * The recorded recipe does not include the candidate's random seed, and the search changes the seed with every mutation. Refitting a best recipe therefore gives a model with the same configuration but a different seed, whose metrics can differ slightly from the cells that selected it. Recipes identical to a fixed recipe are not affected: their default-seed results are in the result files. ## WISP-Funnel: a fast search `WISPSearch("funnel", controller="random" | "evolution")` runs both controllers of the paper's search in one pass on the CIS family: each controller's proposals are ranked on a training-only proxy (at most 4,096 training and 1,024 validation windows, a shared Ridge feature dictionary, cost weight 0), a shortlist of 4 per controller is verified on the full splits with the screened Ridge solver, and each winner is refitted on train+validation ([API.md](API.md#wisp-funnel-wispsearchstrategyfunnel)). It is this package's method and benchmark, not a paper result, and it is not the paper's search protocol. **Benchmark record.** 150 cells: the ten primary streams x 5 folds x 3 seeds (42, 43, 44), the paper's training and validation splits for selection, the test split scored once with each model's own per-person HMM decoder (`decoder="recipe"`). Averages follow the paper rule above (nine primary views). This package reproduces the cells of the record through `WISPSearch(strategy="funnel", decoder="recipe")` with each export's sampling rate and study setting passed to `fit` ([`benchmarks/results/funnel_record_check.md`](../benchmarks/results/funnel_record_check.md)). | Method | Macro-F1 | Accuracy | Balanced acc. | Worst-class F1 | |---|---:|---:|---:|---:| | WISP-Random (paper search) | 70.20 | 77.28 | 70.88 | 28.25 | | WISP-Evolution (paper search) | 69.59 | 76.84 | 70.17 | 27.96 | | **WISP-Funnel, Evolution controller** | **69.30** | 75.93 | 69.97 | 26.99 | | **WISP-Funnel, Random controller** | **69.22** | 76.11 | 70.01 | 26.27 | | WISP-CIS (fixed, no search) | 66.14 | 74.07 | 66.04 | 24.78 | The two funnel models are 0.98 (Random) and 0.29 (Evolution) macro-F1 points below the corresponding paper searches and 3.08 and 3.16 points above WISP-CIS. In paired exact sign-flip tests over the nine views (Holm-adjusted within each metric) none of these macro-F1 differences is significant (adjusted p >= 0.98). **Cost.** Per stream, the median over its 15 cells of the funnel's CPU time and the median over its 15 searches of one paper search's cumulative candidate-fitting time: | Stream | Funnel run, both controllers (core-h) | One WISP-Random search (core-h) | One WISP-Evolution search (core-h) | |---|---:|---:|---:| | ADL-Wrist | 0.022 | 4.04 | 5.37 | | CAPTURE-24 | 1.248 | 54.23 | 78.48 | | DOMINO | 0.242 | 20.02 | 33.31 | | GOTOV | 1.174 | 55.73 | 92.90 | | HANDY | 0.087 | 9.23 | 10.22 | | HARMES left wrist | 1.148 | 53.53 | 60.73 | | HARMES right wrist | 1.248 | 45.59 | 66.83 | | IUWDS | 1.281 | 61.14 | 112.44 | | PAAL-ADL | 0.144 | 11.17 | 15.56 | | WISDM | 1.163 | 50.89 | 65.60 | | **Sum of the ten medians** | **7.76** | **365.57** | **541.45** | So one funnel run, which yields both models, costs about 1/47 of one WISP-Random search and 1/70 of one WISP-Evolution search (sums of the per-stream medians; per stream 1/37 to 1/184 against WISP-Random). What is measured: for the funnel, the whole run's process CPU time (user plus system, both threads): both controllers' proxy search, verification and both refits on train+validation, on two cores of an Intel Xeon Platinum 8570. For the paper searches, the recorded cumulative single-core time of the 329 candidate fits of one search (`results/search_selections.csv`), which leaves out the search's validation and stress-view scoring and its final refit; the comparison therefore favours the paper searches. The largest funnel run took 1.07 h of wall time and at most 12.2 GiB of memory (both controllers in parallel). Source: the record's per-cell RESULT files, summarised in `benchmarks/results/funnel_record_check.md`. ## Cost notes * **Final models are cheap.** Fit on train+validation plus test prediction, core-hours over the ten primary streams (fold 0, seed 42): WISP-Random 0.311, WISP-CIS 0.347, WISP-Evolution 0.712 (`docs/figures/cost_accuracy_plotted_values.csv`, Xeon 8570, one thread). This package's own timings of every fixed family on all twelve streams are in `benchmarks/results/speed_engines.csv` (summary in [SPEED.md](SPEED.md)). * **Selection is the expensive part.** WISPSelect fits each candidate family once on training data before the refit; a search fits up to `budget` candidates and scores each on the validation split and four perturbed copies of it (median 47.5 h / 61.3 h per paper search, above). `WISPSelect`/`WISPSearch` report `selection_seconds_` and `final_fit_seconds_` separately. * **Speed tiers** change only the time, not the recipe: `speed="fast"` affects Ridge readouts (RC, CIS, the RC/CIS members of CSE and ESE, and RC/CIS search candidates) and can change results slightly; paper results correspond to `speed="exact"` only. * **Memory.** Peak host memory of one final model (train+validation fit plus test prediction, one thread, largest of the ten primary streams): WISP-Random 4.7, WISP-Evolution 8.0, WISP-CIS 8.9 GiB (`docs/figures/memory_plotted_values.csv`; per-stream values in `benchmarks/results/memory_final_models.csv`).