Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / docs /RECIPES.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
12.4 kB
# Which recipe, and when to search
The catalog holds only recipes with rows in the saved result files (`results/`). Every
number on this page comes from those files through `benchmarks/catalog_summary.py`
(output: `benchmarks/results/catalog_summary.json`), from this package's own timing benchmark
(`benchmarks/results/speed_engines.csv`, see [SPEED.md](SPEED.md)), or from the cost–accuracy
figure data (`docs/figures/cost_accuracy_plotted_values.csv`); each statement says which.
Paper averages below follow the paper's rule: mean over the 5 folds and 3 seeds (42, 43, 44)
per view, then over the nine primary dataset views (the two HARMES wrist streams form one
view; the nine-channel mHealth and PAMAP2 streams are not in this average). This rule
reproduces the reported means exactly (e.g. WISP-CIS macro-F1 66.1391). They are test-split
results of the paper protocol and are listed for orientation only: none of the advice on this
page was chosen by looking at them (see "How to choose" below).
## How to choose
1. **Start with `"cis"` (WISP-CIS, the default).** It is the recipe that validation most often
picks: WISP-Select5 chose CIS in 106 of its 180 validation selections (SS 44, RC 29, GIS 1;
`models/model_inventory.csv` of the Hugging Face repository), and among the 360 final recipes
of the budget-329 searches, each the validation winner of its search, the fixed WISP-CIS
configuration is the most frequent one (53 of 360; `results/search_selections.csv`). It
needs no search.
2. **Let validation choose among the fixed families with `WISPSelect(menu=5)`** when you have a
validation split of separate participants. It fits five candidates on training data and
refits the winner on train+validation.
3. **Try the search's best recipe of a similar data stream** (`wisp.recipes.best(...)`, table
below), e.g. for similar sensors, window length and activity set.
4. **Search last, and try WISP-Funnel first** (`WISPSearch("funnel")`, [below](#wisp-funnel-a-fast-search)).
One funnel run gives both a Random and an Evolution model for about 1/47 of the
candidate-fitting time of one paper WISP-Random search, within one macro-F1 point of the
full searches on the benchmark. The full searches (`WISPSearch("random" | "evolution")`) are
expensive: on the ten primary streams the median cumulative candidate fitting time of one
paper search (one dataset, fold and seed, 329 candidates) was 47.5 h for WISP-Random and
61.3 h for WISP-Evolution (`results/search_selections.csv`, budget 329, 150 searches each).
The final model of a search is cheap; the cost is in the selection.
Pass `groups` (and `time_index`) at fit and predict time when windows come in sequences: the
default `decoder="adaptive"` then decides on validation data whether the HMM decoder helps
(see [API.md](API.md#the-adaptive-hmm-switch-decoderadaptive)). In the paper, each recipe
carried its own HMM flag; about four in five of the 360 final search recipes kept the HMM
(Random 146/180, Evolution 144/180). The paper numbers on this page use each recipe's own
decoder: reproduce them with `decoder="recipe"`.
## Fixed families
| Recipe | Paper name | Macro-F1 | Balanced acc. | Accuracy | Worst-class F1 |
|---|---|---:|---:|---:|---:|
| `"ss"` | WISP-SS | 58.54 | 59.04 | 73.36 | 21.70 |
| `"rc"` | WISP-RC | 63.44 | 63.59 | 71.46 | 22.87 |
| `"so"` | WISP-SO | 53.59 | 54.19 | 69.31 | 16.75 |
| `"gis"` | WISP-GIS | 55.16 | 55.95 | 70.66 | 19.49 |
| `"cis"` | WISP-CIS | 66.14 | 66.04 | 74.07 | 24.78 |
| `"cse"` | WISP-CSE | 58.74 | 59.10 | 72.86 | 21.22 |
| `"ese"` | WISP-ESE | 57.38 | 57.71 | 71.73 | 19.78 |
| `WISPSelect(5)` | WISP-Select5 | 67.10 | 66.94 | 75.21 | 24.93 |
| `WISPSelect(7)` | WISP-Select7 | 66.69 | 66.65 | 75.01 | 24.65 |
| `WISPSearch("random")` | WISP-Random | 70.20 | 70.88 | 77.28 | 28.25 |
| `WISPSearch("evolution")` | WISP-Evolution | 69.59 | 70.17 | 76.84 | 27.96 |
Source: `results/run_results.csv`, paper average (percent). All fixed families use the HMM decoder
(`decoder="recipe"` reproduces these numbers).
## CIS block subsets
`"cis:R"`, `"cis:I"`, `"cis:S"`, `"cis:RI"`, `"cis:RS"`, `"cis:IS"` (and `"cis:RIS"`, which is
`"cis"`) keep only the named blocks of WISP-CIS: R random convolution, I interval summaries,
S within-window state and spectral features. Each was run with and without the HMM decoder;
switch with `decoder="hmm"` / `decoder="none"`.
| Blocks | Macro-F1, HMM | Macro-F1, direct |
|---|---:|---:|
| R | 63.44 | 57.28 |
| I | 43.89 | 38.63 |
| S | 59.39 | 55.30 |
| RI | 63.59 | 57.65 |
| RS | 66.93 | 60.87 |
| IS | 61.42 | 56.25 |
| RIS (= `"cis"`) | 66.14 | 61.12 |
Source: `results/cis_and_budget_runs.csv` (`kind == "recipe"`), paper average. These are a
controlled block study, not a recommendation: they are test results, so do not pick a subset
from this table for a new study; choose it on your own validation data (e.g. with
`GridSearchCV` over `recipe`).
## Search best recipes
Each search returns its validation winner. The best recipe of a data stream is the final
recipe selected most often in its 30 searches (5 folds x 3 seeds x WISP-Random/WISP-Evolution),
ties broken by the number of folds covered. Only validation decides; no test score is read.
`scripts/build_search_best.py` computes the table from `results/search_selections.csv` and
writes `src/wisp/_data/search_best.json`.
| `wisp.recipes.best(...)` | Recipe | Selected | Folds covered | Same as |
|---|---|---:|---:|---|
| `"adl"` | RC, 256 kernels, Ridge, HMM | 9/30 | 3/5 | — (one of the search's initial templates) |
| `"capture24"` | CIS, 128 kernels, 24 intervals, Ridge, HMM | 8/30 | 4/5 | — |
| `"domino"` | CIS, 384 kernels, 24 intervals, Ridge, HMM | 11/30 | 4/5 | `"cis"` |
| `"gotov"` | same as DOMINO | 20/30 | 5/5 | `"cis"` |
| `"iuwds"` | same as DOMINO | 12/30 | 5/5 | `"cis"` |
| `"harmes_left"` | CIS, 384 kernels, 24 intervals, Ridge, no HMM | 8/30 | 5/5 | `"cis"` with `decoder="none"` |
| `"harmes_right"` | same as HARMES left | 7/30 | 4/5 | `"cis"` with `decoder="none"` |
| `"handy"` | SO, HMM | 9/30 | 3/5 | `"so"` |
| `"wisdm"` | GIS, ExtraTrees, 32 shapelets, HMM | 15/30 | 3/5 | `"gis"` |
| `"paal_adl"` | SS, random forest, segments (5, 10, 20), 8 states, HMM | 8/30 | 4/5 | — (one of the search's initial templates) |
| `"pamap2"` | RC, 256 kernels, Ridge, HMM | 6/30 | 3/5 | — (one of the search's initial templates) |
| `"mhealth"` | SO, HMM | 6/30 | **1/5** | `"so"` |
Notes:
* Eight of the twelve are identical to a fixed recipe or to `"cis"` without the HMM, so they
have results on every fold and seed. The other four (ADL, CAPTURE-24, PAAL-ADL, PAMAP2) have
results only in the searches that selected them.
* The mHealth best recipe comes from a single fold; its support is weak.
* Across all 360 searches the most frequent final recipe is the fixed WISP-CIS (53 times),
one reason it is the package default.
* The recorded recipe does not include the candidate's random seed, and the search changes the
seed with every mutation. Refitting a best recipe therefore gives a model with the same
configuration but a different seed, whose metrics can differ slightly from the cells that
selected it. Recipes identical to a fixed recipe are not affected: their default-seed
results are in the result files.
## WISP-Funnel: a fast search
`WISPSearch("funnel", controller="random" | "evolution")` runs both controllers of the paper's
search in one pass on the CIS family: each controller's proposals are ranked on a
training-only proxy (at most 4,096 training and 1,024 validation windows, a shared Ridge
feature dictionary, cost weight 0), a shortlist of 4 per controller is verified on the full
splits with the screened Ridge solver, and each winner is refitted on train+validation
([API.md](API.md#wisp-funnel-wispsearchstrategyfunnel)). It is this package's method and
benchmark, not a paper result, and it is not the paper's search protocol.
**Benchmark record.** 150 cells: the ten primary streams x 5 folds x 3 seeds (42, 43, 44),
the paper's training and validation splits for selection, the test split scored once with each
model's own per-person HMM decoder (`decoder="recipe"`). Averages follow the paper rule above
(nine primary views). This package reproduces the cells of the record through
`WISPSearch(strategy="funnel", decoder="recipe")` with each export's sampling rate and study
setting passed to `fit` ([`benchmarks/results/funnel_record_check.md`](../benchmarks/results/funnel_record_check.md)).
| Method | Macro-F1 | Accuracy | Balanced acc. | Worst-class F1 |
|---|---:|---:|---:|---:|
| WISP-Random (paper search) | 70.20 | 77.28 | 70.88 | 28.25 |
| WISP-Evolution (paper search) | 69.59 | 76.84 | 70.17 | 27.96 |
| **WISP-Funnel, Evolution controller** | **69.30** | 75.93 | 69.97 | 26.99 |
| **WISP-Funnel, Random controller** | **69.22** | 76.11 | 70.01 | 26.27 |
| WISP-CIS (fixed, no search) | 66.14 | 74.07 | 66.04 | 24.78 |
The two funnel models are 0.98 (Random) and 0.29 (Evolution) macro-F1 points below the
corresponding paper searches and 3.08 and 3.16 points above WISP-CIS. In paired exact
sign-flip tests over the nine views (Holm-adjusted within each metric) none of these
macro-F1 differences is significant (adjusted p >= 0.98).
**Cost.** Per stream, the median over its 15 cells of the funnel's CPU time and the median
over its 15 searches of one paper search's cumulative candidate-fitting time:
| Stream | Funnel run, both controllers (core-h) | One WISP-Random search (core-h) | One WISP-Evolution search (core-h) |
|---|---:|---:|---:|
| ADL-Wrist | 0.022 | 4.04 | 5.37 |
| CAPTURE-24 | 1.248 | 54.23 | 78.48 |
| DOMINO | 0.242 | 20.02 | 33.31 |
| GOTOV | 1.174 | 55.73 | 92.90 |
| HANDY | 0.087 | 9.23 | 10.22 |
| HARMES left wrist | 1.148 | 53.53 | 60.73 |
| HARMES right wrist | 1.248 | 45.59 | 66.83 |
| IUWDS | 1.281 | 61.14 | 112.44 |
| PAAL-ADL | 0.144 | 11.17 | 15.56 |
| WISDM | 1.163 | 50.89 | 65.60 |
| **Sum of the ten medians** | **7.76** | **365.57** | **541.45** |
So one funnel run, which yields both models, costs about 1/47 of one WISP-Random search and
1/70 of one WISP-Evolution search (sums of the per-stream medians; per stream 1/37 to 1/184
against WISP-Random). What is measured: for the funnel, the whole run's process CPU time (user
plus system, both threads): both controllers' proxy search, verification and both refits on
train+validation, on two cores of an Intel Xeon Platinum 8570. For the paper searches, the
recorded cumulative single-core time of the 329 candidate fits of one search
(`results/search_selections.csv`), which leaves out the search's validation and stress-view
scoring and its final refit; the comparison therefore favours the paper searches. The largest
funnel run took 1.07 h of wall time and at most 12.2 GiB of memory (both controllers in
parallel). Source: the record's per-cell RESULT files, summarised in
`benchmarks/results/funnel_record_check.md`.
## Cost notes
* **Final models are cheap.** Fit on train+validation plus test prediction, core-hours over the
ten primary streams (fold 0, seed 42): WISP-Random 0.311, WISP-CIS 0.347, WISP-Evolution
0.712 (`docs/figures/cost_accuracy_plotted_values.csv`, Xeon 8570, one thread). This package's
own timings of every fixed family on all twelve streams are in
`benchmarks/results/speed_engines.csv` (summary in [SPEED.md](SPEED.md)).
* **Selection is the expensive part.** WISPSelect fits each candidate family once on training
data before the refit; a search fits up to `budget` candidates and scores each on the
validation split and four perturbed copies of it (median 47.5 h / 61.3 h per paper search,
above). `WISPSelect`/`WISPSearch` report `selection_seconds_` and `final_fit_seconds_`
separately.
* **Speed tiers** change only the time, not the recipe: `speed="fast"` affects Ridge readouts
(RC, CIS, the RC/CIS members of CSE and ESE, and RC/CIS search candidates) and can change
results slightly; paper results correspond to `speed="exact"` only.
* **Memory.** Peak host memory of one final model (train+validation fit plus test prediction,
one thread, largest of the ten primary streams): WISP-Random 4.7, WISP-Evolution 8.0, WISP-CIS
8.9 GiB (`docs/figures/memory_plotted_values.csv`; per-stream values in
`benchmarks/results/memory_final_models.csv`).