Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
File size: 12,383 Bytes
4731b11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93a9282
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4731b11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93a9282
 
4731b11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93a9282
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4731b11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19383e1
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
# Which recipe, and when to search

The catalog holds only recipes with rows in the saved result files (`results/`). Every
number on this page comes from those files through `benchmarks/catalog_summary.py`
(output: `benchmarks/results/catalog_summary.json`), from this package's own timing benchmark
(`benchmarks/results/speed_engines.csv`, see [SPEED.md](SPEED.md)), or from the cost–accuracy
figure data (`docs/figures/cost_accuracy_plotted_values.csv`); each statement says which.

Paper averages below follow the paper's rule: mean over the 5 folds and 3 seeds (42, 43, 44)
per view, then over the nine primary dataset views (the two HARMES wrist streams form one
view; the nine-channel mHealth and PAMAP2 streams are not in this average). This rule
reproduces the reported means exactly (e.g. WISP-CIS macro-F1 66.1391). They are test-split
results of the paper protocol and are listed for orientation only: none of the advice on this
page was chosen by looking at them (see "How to choose" below).

## How to choose

1. **Start with `"cis"` (WISP-CIS, the default).** It is the recipe that validation most often
   picks: WISP-Select5 chose CIS in 106 of its 180 validation selections (SS 44, RC 29, GIS 1;
   `models/model_inventory.csv` of the Hugging Face repository), and among the 360 final recipes
   of the budget-329 searches, each the validation winner of its search, the fixed WISP-CIS
   configuration is the most frequent one (53 of 360; `results/search_selections.csv`). It
   needs no search.
2. **Let validation choose among the fixed families with `WISPSelect(menu=5)`** when you have a
   validation split of separate participants. It fits five candidates on training data and
   refits the winner on train+validation.
3. **Try the search's best recipe of a similar data stream** (`wisp.recipes.best(...)`, table
   below), e.g. for similar sensors, window length and activity set.
4. **Search last, and try WISP-Funnel first** (`WISPSearch("funnel")`, [below](#wisp-funnel-a-fast-search)).
   One funnel run gives both a Random and an Evolution model for about 1/47 of the
   candidate-fitting time of one paper WISP-Random search, within one macro-F1 point of the
   full searches on the benchmark. The full searches (`WISPSearch("random" | "evolution")`) are
   expensive: on the ten primary streams the median cumulative candidate fitting time of one
   paper search (one dataset, fold and seed, 329 candidates) was 47.5 h for WISP-Random and
   61.3 h for WISP-Evolution (`results/search_selections.csv`, budget 329, 150 searches each).
   The final model of a search is cheap; the cost is in the selection.

Pass `groups` (and `time_index`) at fit and predict time when windows come in sequences: the
default `decoder="adaptive"` then decides on validation data whether the HMM decoder helps
(see [API.md](API.md#the-adaptive-hmm-switch-decoderadaptive)). In the paper, each recipe
carried its own HMM flag; about four in five of the 360 final search recipes kept the HMM
(Random 146/180, Evolution 144/180). The paper numbers on this page use each recipe's own
decoder: reproduce them with `decoder="recipe"`.

## Fixed families

| Recipe | Paper name | Macro-F1 | Balanced acc. | Accuracy | Worst-class F1 |
|---|---|---:|---:|---:|---:|
| `"ss"` | WISP-SS | 58.54 | 59.04 | 73.36 | 21.70 |
| `"rc"` | WISP-RC | 63.44 | 63.59 | 71.46 | 22.87 |
| `"so"` | WISP-SO | 53.59 | 54.19 | 69.31 | 16.75 |
| `"gis"` | WISP-GIS | 55.16 | 55.95 | 70.66 | 19.49 |
| `"cis"` | WISP-CIS | 66.14 | 66.04 | 74.07 | 24.78 |
| `"cse"` | WISP-CSE | 58.74 | 59.10 | 72.86 | 21.22 |
| `"ese"` | WISP-ESE | 57.38 | 57.71 | 71.73 | 19.78 |
| `WISPSelect(5)` | WISP-Select5 | 67.10 | 66.94 | 75.21 | 24.93 |
| `WISPSelect(7)` | WISP-Select7 | 66.69 | 66.65 | 75.01 | 24.65 |
| `WISPSearch("random")` | WISP-Random | 70.20 | 70.88 | 77.28 | 28.25 |
| `WISPSearch("evolution")` | WISP-Evolution | 69.59 | 70.17 | 76.84 | 27.96 |

Source: `results/run_results.csv`, paper average (percent). All fixed families use the HMM decoder
(`decoder="recipe"` reproduces these numbers).

## CIS block subsets

`"cis:R"`, `"cis:I"`, `"cis:S"`, `"cis:RI"`, `"cis:RS"`, `"cis:IS"` (and `"cis:RIS"`, which is
`"cis"`) keep only the named blocks of WISP-CIS: R random convolution, I interval summaries,
S within-window state and spectral features. Each was run with and without the HMM decoder;
switch with `decoder="hmm"` / `decoder="none"`.

| Blocks | Macro-F1, HMM | Macro-F1, direct |
|---|---:|---:|
| R | 63.44 | 57.28 |
| I | 43.89 | 38.63 |
| S | 59.39 | 55.30 |
| RI | 63.59 | 57.65 |
| RS | 66.93 | 60.87 |
| IS | 61.42 | 56.25 |
| RIS (= `"cis"`) | 66.14 | 61.12 |

Source: `results/cis_and_budget_runs.csv` (`kind == "recipe"`), paper average. These are a
controlled block study, not a recommendation: they are test results, so do not pick a subset
from this table for a new study; choose it on your own validation data (e.g. with
`GridSearchCV` over `recipe`).

## Search best recipes

Each search returns its validation winner. The best recipe of a data stream is the final
recipe selected most often in its 30 searches (5 folds x 3 seeds x WISP-Random/WISP-Evolution),
ties broken by the number of folds covered. Only validation decides; no test score is read.
`scripts/build_search_best.py` computes the table from `results/search_selections.csv` and
writes `src/wisp/_data/search_best.json`.

| `wisp.recipes.best(...)` | Recipe | Selected | Folds covered | Same as |
|---|---|---:|---:|---|
| `"adl"` | RC, 256 kernels, Ridge, HMM | 9/30 | 3/5 | — (one of the search's initial templates) |
| `"capture24"` | CIS, 128 kernels, 24 intervals, Ridge, HMM | 8/30 | 4/5 | — |
| `"domino"` | CIS, 384 kernels, 24 intervals, Ridge, HMM | 11/30 | 4/5 | `"cis"` |
| `"gotov"` | same as DOMINO | 20/30 | 5/5 | `"cis"` |
| `"iuwds"` | same as DOMINO | 12/30 | 5/5 | `"cis"` |
| `"harmes_left"` | CIS, 384 kernels, 24 intervals, Ridge, no HMM | 8/30 | 5/5 | `"cis"` with `decoder="none"` |
| `"harmes_right"` | same as HARMES left | 7/30 | 4/5 | `"cis"` with `decoder="none"` |
| `"handy"` | SO, HMM | 9/30 | 3/5 | `"so"` |
| `"wisdm"` | GIS, ExtraTrees, 32 shapelets, HMM | 15/30 | 3/5 | `"gis"` |
| `"paal_adl"` | SS, random forest, segments (5, 10, 20), 8 states, HMM | 8/30 | 4/5 | — (one of the search's initial templates) |
| `"pamap2"` | RC, 256 kernels, Ridge, HMM | 6/30 | 3/5 | — (one of the search's initial templates) |
| `"mhealth"` | SO, HMM | 6/30 | **1/5** | `"so"` |

Notes:

* Eight of the twelve are identical to a fixed recipe or to `"cis"` without the HMM, so they
  have results on every fold and seed. The other four (ADL, CAPTURE-24, PAAL-ADL, PAMAP2) have
  results only in the searches that selected them.
* The mHealth best recipe comes from a single fold; its support is weak.
* Across all 360 searches the most frequent final recipe is the fixed WISP-CIS (53 times),
  one reason it is the package default.
* The recorded recipe does not include the candidate's random seed, and the search changes the
  seed with every mutation. Refitting a best recipe therefore gives a model with the same
  configuration but a different seed, whose metrics can differ slightly from the cells that
  selected it. Recipes identical to a fixed recipe are not affected: their default-seed
  results are in the result files.

## WISP-Funnel: a fast search

`WISPSearch("funnel", controller="random" | "evolution")` runs both controllers of the paper's
search in one pass on the CIS family: each controller's proposals are ranked on a
training-only proxy (at most 4,096 training and 1,024 validation windows, a shared Ridge
feature dictionary, cost weight 0), a shortlist of 4 per controller is verified on the full
splits with the screened Ridge solver, and each winner is refitted on train+validation
([API.md](API.md#wisp-funnel-wispsearchstrategyfunnel)). It is this package's method and
benchmark, not a paper result, and it is not the paper's search protocol.

**Benchmark record.** 150 cells: the ten primary streams x 5 folds x 3 seeds (42, 43, 44),
the paper's training and validation splits for selection, the test split scored once with each
model's own per-person HMM decoder (`decoder="recipe"`). Averages follow the paper rule above
(nine primary views). This package reproduces the cells of the record through
`WISPSearch(strategy="funnel", decoder="recipe")` with each export's sampling rate and study
setting passed to `fit` ([`benchmarks/results/funnel_record_check.md`](../benchmarks/results/funnel_record_check.md)).

| Method | Macro-F1 | Accuracy | Balanced acc. | Worst-class F1 |
|---|---:|---:|---:|---:|
| WISP-Random (paper search) | 70.20 | 77.28 | 70.88 | 28.25 |
| WISP-Evolution (paper search) | 69.59 | 76.84 | 70.17 | 27.96 |
| **WISP-Funnel, Evolution controller** | **69.30** | 75.93 | 69.97 | 26.99 |
| **WISP-Funnel, Random controller** | **69.22** | 76.11 | 70.01 | 26.27 |
| WISP-CIS (fixed, no search) | 66.14 | 74.07 | 66.04 | 24.78 |

The two funnel models are 0.98 (Random) and 0.29 (Evolution) macro-F1 points below the
corresponding paper searches and 3.08 and 3.16 points above WISP-CIS. In paired exact
sign-flip tests over the nine views (Holm-adjusted within each metric) none of these
macro-F1 differences is significant (adjusted p >= 0.98).

**Cost.** Per stream, the median over its 15 cells of the funnel's CPU time and the median
over its 15 searches of one paper search's cumulative candidate-fitting time:

| Stream | Funnel run, both controllers (core-h) | One WISP-Random search (core-h) | One WISP-Evolution search (core-h) |
|---|---:|---:|---:|
| ADL-Wrist | 0.022 | 4.04 | 5.37 |
| CAPTURE-24 | 1.248 | 54.23 | 78.48 |
| DOMINO | 0.242 | 20.02 | 33.31 |
| GOTOV | 1.174 | 55.73 | 92.90 |
| HANDY | 0.087 | 9.23 | 10.22 |
| HARMES left wrist | 1.148 | 53.53 | 60.73 |
| HARMES right wrist | 1.248 | 45.59 | 66.83 |
| IUWDS | 1.281 | 61.14 | 112.44 |
| PAAL-ADL | 0.144 | 11.17 | 15.56 |
| WISDM | 1.163 | 50.89 | 65.60 |
| **Sum of the ten medians** | **7.76** | **365.57** | **541.45** |

So one funnel run, which yields both models, costs about 1/47 of one WISP-Random search and
1/70 of one WISP-Evolution search (sums of the per-stream medians; per stream 1/37 to 1/184
against WISP-Random). What is measured: for the funnel, the whole run's process CPU time (user
plus system, both threads): both controllers' proxy search, verification and both refits on
train+validation, on two cores of an Intel Xeon Platinum 8570. For the paper searches, the
recorded cumulative single-core time of the 329 candidate fits of one search
(`results/search_selections.csv`), which leaves out the search's validation and stress-view
scoring and its final refit; the comparison therefore favours the paper searches. The largest
funnel run took 1.07 h of wall time and at most 12.2 GiB of memory (both controllers in
parallel). Source: the record's per-cell RESULT files, summarised in
`benchmarks/results/funnel_record_check.md`.

## Cost notes

* **Final models are cheap.** Fit on train+validation plus test prediction, core-hours over the
  ten primary streams (fold 0, seed 42): WISP-Random 0.311, WISP-CIS 0.347, WISP-Evolution
  0.712 (`docs/figures/cost_accuracy_plotted_values.csv`, Xeon 8570, one thread). This package's
  own timings of every fixed family on all twelve streams are in
  `benchmarks/results/speed_engines.csv` (summary in [SPEED.md](SPEED.md)).
* **Selection is the expensive part.** WISPSelect fits each candidate family once on training
  data before the refit; a search fits up to `budget` candidates and scores each on the
  validation split and four perturbed copies of it (median 47.5 h / 61.3 h per paper search,
  above). `WISPSelect`/`WISPSearch` report `selection_seconds_` and `final_fit_seconds_`
  separately.
* **Speed tiers** change only the time, not the recipe: `speed="fast"` affects Ridge readouts
  (RC, CIS, the RC/CIS members of CSE and ESE, and RC/CIS search candidates) and can change
  results slightly; paper results correspond to `speed="exact"` only.
* **Memory.** Peak host memory of one final model (train+validation fit plus test prediction,
  one thread, largest of the ten primary streams): WISP-Random 4.7, WISP-Evolution 8.0, WISP-CIS
  8.9 GiB (`docs/figures/memory_plotted_values.csv`; per-stream values in
  `benchmarks/results/memory_final_models.csv`).