Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / benchmarks /results /adaptive_decoder.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
6.57 kB
# `decoder="adaptive"` on the 150 WristHARBench cells
**This package's benchmark, not a paper result.** The 0.4.0 default decoder on the ten primary streams x 5 folds x seeds 42, 43, 44, with the paper's validation split as decision data and the test split scored once, against the paper's fixed-decoder results (`results/run_results.csv`; for WISP-Funnel, the funnel benchmark record, reproduced with `decoder="recipe"`). Fixed families: `WISPClassifier(<family>).fit(train, validation=valid)`; WISPSelect: `WISPSelect(5 | 7)`; WISP-Random/WISP-Evolution: the adaptive step of `WISPSearch` on the recipe each paper search selected (budget 329), refitted with its candidate seed (the searches were not rerun); WISP-Funnel: `WISPSearch("funnel")`. One BLAS thread (four for the five retried cells in FAILURES.md, entries 14-15). Slurm: 61882594 (fixed, Select, saved winners), 61884519/61884764/61885092 (saved-winner reruns), 61884684-61884686 and 61885046 (BLAS-4 retries), 61885547 (WISP-Funnel). Script: `benchmarks/adaptive_decoder.py` and `benchmarks/funnel_cells.py`.
## Nine-view averages (paper rule)
| Method | Cells | HMM kept | Decoder changed | Macro-F1 adaptive / paper | Accuracy | Balanced acc. | Worst-class F1 | Extra time |
|---|---:|---:|---:|---|---|---|---|---:|
| WISP-SS | 150 | 77 | 73 | 61.45 / 58.54 (+2.91) | 73.47 / 73.36 (+0.11) | 61.97 / 59.04 (+2.93) | 25.94 / 21.70 (+4.25) | 1.11 h (88%) |
| WISP-RC | 150 | 105 | 45 | 65.41 / 63.44 (+1.96) | 72.66 / 71.46 (+1.20) | 66.78 / 63.59 (+3.20) | 23.67 / 22.87 (+0.79) | 0.68 h (91%) |
| WISP-SO | 150 | 79 | 71 | 55.70 / 53.59 (+2.11) | 69.15 / 69.31 (-0.15) | 56.44 / 54.19 (+2.25) | 21.00 / 16.75 (+4.25) | 1.10 h (88%) |
| WISP-GIS | 150 | 76 | 74 | 58.10 / 55.16 (+2.94) | 70.56 / 70.66 (-0.10) | 58.42 / 55.95 (+2.47) | 24.11 / 19.49 (+4.62) | 1.53 h (88%) |
| WISP-CIS | 150 | 111 | 39 | 68.54 / 66.14 (+2.40) | 75.41 / 74.07 (+1.34) | 69.37 / 66.04 (+3.33) | 26.78 / 24.78 (+2.00) | 2.24 h (90%) |
| WISP-CSE | 150 | 75 | 75 | 62.44 / 58.74 (+3.70) | 73.43 / 72.86 (+0.57) | 62.94 / 59.10 (+3.84) | 25.75 / 21.22 (+4.53) | 2.58 h (88%) |
| WISP-ESE | 150 | 70 | 80 | 60.77 / 57.38 (+3.39) | 71.91 / 71.73 (+0.17) | 61.01 / 57.71 (+3.30) | 24.52 / 19.78 (+4.74) | 3.93 h (89%) |
| WISP-Select5 | 150 | 112 | 38 | 69.27 / 67.10 (+2.17) | 76.39 / 75.21 (+1.18) | 69.81 / 66.94 (+2.87) | 26.80 / 24.93 (+1.86) | 0.64 h (31%) |
| WISP-Select7 | 150 | 112 | 38 | 68.86 / 66.69 (+2.17) | 76.18 / 75.01 (+1.18) | 69.52 / 66.65 (+2.87) | 26.51 / 24.65 (+1.86) | 0.65 h (29%) |
| WISP-Random (saved winner) | 150 | 120 | 0 | 70.20 / 70.20 (+0.00) | 77.28 / 77.28 (+0.00) | 70.88 / 70.88 (+0.00) | 28.25 / 28.25 (+0.00) | 2.38 h (89%) |
| WISP-Evolution (saved winner) | 150 | 120 | 0 | 69.59 / 69.59 (+0.00) | 76.84 / 76.84 (+0.00) | 70.17 / 70.17 (+0.00) | 27.96 / 27.96 (+0.00) | 2.06 h (89%) |
| WISP-Funnel, Random | 150 | 116 | 4 | 69.20 / 69.22 (-0.02) | 76.37 / 76.11 (+0.26) | 70.18 / 70.01 (+0.16) | 26.18 / 26.27 (-0.09) | 6.26 h (78%) |
| WISP-Funnel, Evolution | 150 | 119 | 5 | 69.76 / 69.30 (+0.46) | 76.16 / 75.93 (+0.23) | 70.10 / 69.97 (+0.13) | 26.77 / 26.99 (-0.23) | 7.14 h (85%) |
"HMM kept": cells whose final model decodes with the HMM. "Decoder changed": cells where the switch chose another decoder than the recipe's own. Extra time: the decision (training-only fit and validation decoding; for WISPSelect only the validation prediction, since the selector keeps its training fits) plus any refit caused by a change of decoder, summed over the cells, and in brackets relative to the final fits' time.
## Cells where the decoder changed: test change (adaptive minus paper, points)
| Method | Changed | Macro-F1 up / down | Mean | Worst | Best | Balanced acc. mean / worst | Accuracy mean / worst |
|---|---:|---|---:|---:|---:|---|---|
| WISP-SS | 73 | 67 / 6 | +8.03 | -27.72 | +16.22 | +7.62 / -3.01 | +1.68 / -7.43 |
| WISP-RC | 45 | 37 / 8 | +12.62 | -23.49 | +27.12 | +17.84 / -11.93 | +7.98 / -22.79 |
| WISP-SO | 71 | 56 / 15 | +6.51 | -27.30 | +15.86 | +6.27 / -4.33 | +1.25 / -9.41 |
| WISP-GIS | 74 | 58 / 16 | +8.07 | -24.85 | +19.29 | +6.67 / -8.94 | +1.42 / -8.91 |
| WISP-CIS | 39 | 36 / 3 | +16.18 | -10.79 | +25.38 | +21.15 / -1.07 | +9.84 / -14.49 |
| WISP-CSE | 75 | 66 / 9 | +10.06 | -23.16 | +20.63 | +9.74 / -5.79 | +3.17 / -8.69 |
| WISP-ESE | 80 | 68 / 12 | +8.96 | -23.47 | +22.26 | +8.22 / -7.29 | +2.44 / -9.64 |
| WISP-Select5 | 38 | 36 / 2 | +14.97 | -23.49 | +21.87 | +18.64 / -11.93 | +8.74 / -22.79 |
| WISP-Select7 | 38 | 36 / 2 | +14.97 | -23.49 | +21.87 | +18.64 / -11.93 | +8.74 / -22.79 |
| WISP-Random (saved winner) | 0 | | | | | | |
| WISP-Evolution (saved winner) | 0 | | | | | | |
| WISP-Funnel, Random | 4 | 3 / 1 | -0.68 | -23.91 | +11.46 | +5.46 / +0.21 | +8.73 / +6.91 |
| WISP-Funnel, Evolution | 5 | 5 / 0 | +12.54 | +3.80 | +25.30 | +3.46 / -9.70 | +6.22 / -9.07 |
## HMM kept, by stream (of 15 cells; changed decoders in brackets)
| Method | ADL | CAP24 | DOMINO | GOTOV | HANDY | HARMES-L | HARMES-R | IUWDS | PAAL | WISDM |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| WISP-SS | 6 (9) | 0 (15) | 11 (4) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 0 (15) | 15 (0) | 15 (0) |
| WISP-RC | 9 (6) | 14 (1) | 13 (2) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 13 (2) | 11 (4) | 15 (0) |
| WISP-SO | 7 (8) | 0 (15) | 3 (12) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 9 (6) | 15 (0) | 15 (0) |
| WISP-GIS | 6 (9) | 0 (15) | 5 (10) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 5 (10) | 15 (0) | 15 (0) |
| WISP-CIS | 10 (5) | 15 (0) | 12 (3) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 14 (1) | 15 (0) | 15 (0) |
| WISP-CSE | 6 (9) | 0 (15) | 9 (6) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 0 (15) | 15 (0) | 15 (0) |
| WISP-ESE | 6 (9) | 0 (15) | 4 (11) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 0 (15) | 15 (0) | 15 (0) |
| WISP-Select5 | 10 (5) | 15 (0) | 13 (2) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 14 (1) | 15 (0) | 15 (0) |
| WISP-Select7 | 10 (5) | 15 (0) | 13 (2) | 15 (0) | 15 (0) | 0 (15) | 0 (15) | 14 (1) | 15 (0) | 15 (0) |
| WISP-Random (saved winner) | 15 (0) | 15 (0) | 15 (0) | 15 (0) | 15 (0) | 0 (0) | 0 (0) | 15 (0) | 15 (0) | 15 (0) |
| WISP-Evolution (saved winner) | 15 (0) | 13 (0) | 15 (0) | 15 (0) | 15 (0) | 1 (0) | 1 (0) | 15 (0) | 15 (0) | 15 (0) |
| WISP-Funnel, Random | 11 (1) | 15 (2) | 15 (0) | 15 (1) | 15 (0) | 0 (0) | 0 (0) | 15 (0) | 15 (0) | 15 (0) |
| WISP-Funnel, Evolution | 14 (1) | 15 (2) | 15 (0) | 15 (2) | 15 (0) | 0 (0) | 0 (0) | 15 (0) | 15 (0) | 15 (0) |
Per-cell rows: `adaptive_decoder.csv`.