Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / docs /RECIPES.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
12.4 kB

Which recipe, and when to search

The catalog holds only recipes with rows in the saved result files (results/). Every number on this page comes from those files through benchmarks/catalog_summary.py (output: benchmarks/results/catalog_summary.json), from this package's own timing benchmark (benchmarks/results/speed_engines.csv, see SPEED.md), or from the cost–accuracy figure data (docs/figures/cost_accuracy_plotted_values.csv); each statement says which.

Paper averages below follow the paper's rule: mean over the 5 folds and 3 seeds (42, 43, 44) per view, then over the nine primary dataset views (the two HARMES wrist streams form one view; the nine-channel mHealth and PAMAP2 streams are not in this average). This rule reproduces the reported means exactly (e.g. WISP-CIS macro-F1 66.1391). They are test-split results of the paper protocol and are listed for orientation only: none of the advice on this page was chosen by looking at them (see "How to choose" below).

How to choose

  1. Start with "cis" (WISP-CIS, the default). It is the recipe that validation most often picks: WISP-Select5 chose CIS in 106 of its 180 validation selections (SS 44, RC 29, GIS 1; models/model_inventory.csv of the Hugging Face repository), and among the 360 final recipes of the budget-329 searches, each the validation winner of its search, the fixed WISP-CIS configuration is the most frequent one (53 of 360; results/search_selections.csv). It needs no search.
  2. Let validation choose among the fixed families with WISPSelect(menu=5) when you have a validation split of separate participants. It fits five candidates on training data and refits the winner on train+validation.
  3. Try the search's best recipe of a similar data stream (wisp.recipes.best(...), table below), e.g. for similar sensors, window length and activity set.
  4. Search last, and try WISP-Funnel first (WISPSearch("funnel"), below). One funnel run gives both a Random and an Evolution model for about 1/47 of the candidate-fitting time of one paper WISP-Random search, within one macro-F1 point of the full searches on the benchmark. The full searches (WISPSearch("random" | "evolution")) are expensive: on the ten primary streams the median cumulative candidate fitting time of one paper search (one dataset, fold and seed, 329 candidates) was 47.5 h for WISP-Random and 61.3 h for WISP-Evolution (results/search_selections.csv, budget 329, 150 searches each). The final model of a search is cheap; the cost is in the selection.

Pass groups (and time_index) at fit and predict time when windows come in sequences: the default decoder="adaptive" then decides on validation data whether the HMM decoder helps (see API.md). In the paper, each recipe carried its own HMM flag; about four in five of the 360 final search recipes kept the HMM (Random 146/180, Evolution 144/180). The paper numbers on this page use each recipe's own decoder: reproduce them with decoder="recipe".

Fixed families

Recipe Paper name Macro-F1 Balanced acc. Accuracy Worst-class F1
"ss" WISP-SS 58.54 59.04 73.36 21.70
"rc" WISP-RC 63.44 63.59 71.46 22.87
"so" WISP-SO 53.59 54.19 69.31 16.75
"gis" WISP-GIS 55.16 55.95 70.66 19.49
"cis" WISP-CIS 66.14 66.04 74.07 24.78
"cse" WISP-CSE 58.74 59.10 72.86 21.22
"ese" WISP-ESE 57.38 57.71 71.73 19.78
WISPSelect(5) WISP-Select5 67.10 66.94 75.21 24.93
WISPSelect(7) WISP-Select7 66.69 66.65 75.01 24.65
WISPSearch("random") WISP-Random 70.20 70.88 77.28 28.25
WISPSearch("evolution") WISP-Evolution 69.59 70.17 76.84 27.96

Source: results/run_results.csv, paper average (percent). All fixed families use the HMM decoder (decoder="recipe" reproduces these numbers).

CIS block subsets

"cis:R", "cis:I", "cis:S", "cis:RI", "cis:RS", "cis:IS" (and "cis:RIS", which is "cis") keep only the named blocks of WISP-CIS: R random convolution, I interval summaries, S within-window state and spectral features. Each was run with and without the HMM decoder; switch with decoder="hmm" / decoder="none".

Blocks Macro-F1, HMM Macro-F1, direct
R 63.44 57.28
I 43.89 38.63
S 59.39 55.30
RI 63.59 57.65
RS 66.93 60.87
IS 61.42 56.25
RIS (= "cis") 66.14 61.12

Source: results/cis_and_budget_runs.csv (kind == "recipe"), paper average. These are a controlled block study, not a recommendation: they are test results, so do not pick a subset from this table for a new study; choose it on your own validation data (e.g. with GridSearchCV over recipe).

Search best recipes

Each search returns its validation winner. The best recipe of a data stream is the final recipe selected most often in its 30 searches (5 folds x 3 seeds x WISP-Random/WISP-Evolution), ties broken by the number of folds covered. Only validation decides; no test score is read. scripts/build_search_best.py computes the table from results/search_selections.csv and writes src/wisp/_data/search_best.json.

wisp.recipes.best(...) Recipe Selected Folds covered Same as
"adl" RC, 256 kernels, Ridge, HMM 9/30 3/5 — (one of the search's initial templates)
"capture24" CIS, 128 kernels, 24 intervals, Ridge, HMM 8/30 4/5 —
"domino" CIS, 384 kernels, 24 intervals, Ridge, HMM 11/30 4/5 "cis"
"gotov" same as DOMINO 20/30 5/5 "cis"
"iuwds" same as DOMINO 12/30 5/5 "cis"
"harmes_left" CIS, 384 kernels, 24 intervals, Ridge, no HMM 8/30 5/5 "cis" with decoder="none"
"harmes_right" same as HARMES left 7/30 4/5 "cis" with decoder="none"
"handy" SO, HMM 9/30 3/5 "so"
"wisdm" GIS, ExtraTrees, 32 shapelets, HMM 15/30 3/5 "gis"
"paal_adl" SS, random forest, segments (5, 10, 20), 8 states, HMM 8/30 4/5 — (one of the search's initial templates)
"pamap2" RC, 256 kernels, Ridge, HMM 6/30 3/5 — (one of the search's initial templates)
"mhealth" SO, HMM 6/30 1/5 "so"

Notes:

  • Eight of the twelve are identical to a fixed recipe or to "cis" without the HMM, so they have results on every fold and seed. The other four (ADL, CAPTURE-24, PAAL-ADL, PAMAP2) have results only in the searches that selected them.
  • The mHealth best recipe comes from a single fold; its support is weak.
  • Across all 360 searches the most frequent final recipe is the fixed WISP-CIS (53 times), one reason it is the package default.
  • The recorded recipe does not include the candidate's random seed, and the search changes the seed with every mutation. Refitting a best recipe therefore gives a model with the same configuration but a different seed, whose metrics can differ slightly from the cells that selected it. Recipes identical to a fixed recipe are not affected: their default-seed results are in the result files.

WISP-Funnel: a fast search

WISPSearch("funnel", controller="random" | "evolution") runs both controllers of the paper's search in one pass on the CIS family: each controller's proposals are ranked on a training-only proxy (at most 4,096 training and 1,024 validation windows, a shared Ridge feature dictionary, cost weight 0), a shortlist of 4 per controller is verified on the full splits with the screened Ridge solver, and each winner is refitted on train+validation (API.md). It is this package's method and benchmark, not a paper result, and it is not the paper's search protocol.

Benchmark record. 150 cells: the ten primary streams x 5 folds x 3 seeds (42, 43, 44), the paper's training and validation splits for selection, the test split scored once with each model's own per-person HMM decoder (decoder="recipe"). Averages follow the paper rule above (nine primary views). This package reproduces the cells of the record through WISPSearch(strategy="funnel", decoder="recipe") with each export's sampling rate and study setting passed to fit (benchmarks/results/funnel_record_check.md).

Method Macro-F1 Accuracy Balanced acc. Worst-class F1
WISP-Random (paper search) 70.20 77.28 70.88 28.25
WISP-Evolution (paper search) 69.59 76.84 70.17 27.96
WISP-Funnel, Evolution controller 69.30 75.93 69.97 26.99
WISP-Funnel, Random controller 69.22 76.11 70.01 26.27
WISP-CIS (fixed, no search) 66.14 74.07 66.04 24.78

The two funnel models are 0.98 (Random) and 0.29 (Evolution) macro-F1 points below the corresponding paper searches and 3.08 and 3.16 points above WISP-CIS. In paired exact sign-flip tests over the nine views (Holm-adjusted within each metric) none of these macro-F1 differences is significant (adjusted p >= 0.98).

Cost. Per stream, the median over its 15 cells of the funnel's CPU time and the median over its 15 searches of one paper search's cumulative candidate-fitting time:

Stream Funnel run, both controllers (core-h) One WISP-Random search (core-h) One WISP-Evolution search (core-h)
ADL-Wrist 0.022 4.04 5.37
CAPTURE-24 1.248 54.23 78.48
DOMINO 0.242 20.02 33.31
GOTOV 1.174 55.73 92.90
HANDY 0.087 9.23 10.22
HARMES left wrist 1.148 53.53 60.73
HARMES right wrist 1.248 45.59 66.83
IUWDS 1.281 61.14 112.44
PAAL-ADL 0.144 11.17 15.56
WISDM 1.163 50.89 65.60
Sum of the ten medians 7.76 365.57 541.45

So one funnel run, which yields both models, costs about 1/47 of one WISP-Random search and 1/70 of one WISP-Evolution search (sums of the per-stream medians; per stream 1/37 to 1/184 against WISP-Random). What is measured: for the funnel, the whole run's process CPU time (user plus system, both threads): both controllers' proxy search, verification and both refits on train+validation, on two cores of an Intel Xeon Platinum 8570. For the paper searches, the recorded cumulative single-core time of the 329 candidate fits of one search (results/search_selections.csv), which leaves out the search's validation and stress-view scoring and its final refit; the comparison therefore favours the paper searches. The largest funnel run took 1.07 h of wall time and at most 12.2 GiB of memory (both controllers in parallel). Source: the record's per-cell RESULT files, summarised in benchmarks/results/funnel_record_check.md.

Cost notes

  • Final models are cheap. Fit on train+validation plus test prediction, core-hours over the ten primary streams (fold 0, seed 42): WISP-Random 0.311, WISP-CIS 0.347, WISP-Evolution 0.712 (docs/figures/cost_accuracy_plotted_values.csv, Xeon 8570, one thread). This package's own timings of every fixed family on all twelve streams are in benchmarks/results/speed_engines.csv (summary in SPEED.md).
  • Selection is the expensive part. WISPSelect fits each candidate family once on training data before the refit; a search fits up to budget candidates and scores each on the validation split and four perturbed copies of it (median 47.5 h / 61.3 h per paper search, above). WISPSelect/WISPSearch report selection_seconds_ and final_fit_seconds_ separately.
  • Speed tiers change only the time, not the recipe: speed="fast" affects Ridge readouts (RC, CIS, the RC/CIS members of CSE and ESE, and RC/CIS search candidates) and can change results slightly; paper results correspond to speed="exact" only.
  • Memory. Peak host memory of one final model (train+validation fit plus test prediction, one thread, largest of the ten primary streams): WISP-Random 4.7, WISP-Evolution 8.0, WISP-CIS 8.9 GiB (docs/figures/memory_plotted_values.csv; per-stream values in benchmarks/results/memory_final_models.csv).