Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / benchmarks /results /adaptive_decoder.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
6.57 kB

decoder="adaptive" on the 150 WristHARBench cells

This package's benchmark, not a paper result. The 0.4.0 default decoder on the ten primary streams x 5 folds x seeds 42, 43, 44, with the paper's validation split as decision data and the test split scored once, against the paper's fixed-decoder results (results/run_results.csv; for WISP-Funnel, the funnel benchmark record, reproduced with decoder="recipe"). Fixed families: WISPClassifier(<family>).fit(train, validation=valid); WISPSelect: WISPSelect(5 | 7); WISP-Random/WISP-Evolution: the adaptive step of WISPSearch on the recipe each paper search selected (budget 329), refitted with its candidate seed (the searches were not rerun); WISP-Funnel: WISPSearch("funnel"). One BLAS thread (four for the five retried cells in FAILURES.md, entries 14-15). Slurm: 61882594 (fixed, Select, saved winners), 61884519/61884764/61885092 (saved-winner reruns), 61884684-61884686 and 61885046 (BLAS-4 retries), 61885547 (WISP-Funnel). Script: benchmarks/adaptive_decoder.py and benchmarks/funnel_cells.py.

Nine-view averages (paper rule)

Method Cells HMM kept Decoder changed Macro-F1 adaptive / paper Accuracy Balanced acc. Worst-class F1 Extra time
WISP-SS 150 77 73 61.45 / 58.54 (+2.91) 73.47 / 73.36 (+0.11) 61.97 / 59.04 (+2.93) 25.94 / 21.70 (+4.25) 1.11 h (88%)
WISP-RC 150 105 45 65.41 / 63.44 (+1.96) 72.66 / 71.46 (+1.20) 66.78 / 63.59 (+3.20) 23.67 / 22.87 (+0.79) 0.68 h (91%)
WISP-SO 150 79 71 55.70 / 53.59 (+2.11) 69.15 / 69.31 (-0.15) 56.44 / 54.19 (+2.25) 21.00 / 16.75 (+4.25) 1.10 h (88%)
WISP-GIS 150 76 74 58.10 / 55.16 (+2.94) 70.56 / 70.66 (-0.10) 58.42 / 55.95 (+2.47) 24.11 / 19.49 (+4.62) 1.53 h (88%)
WISP-CIS 150 111 39 68.54 / 66.14 (+2.40) 75.41 / 74.07 (+1.34) 69.37 / 66.04 (+3.33) 26.78 / 24.78 (+2.00) 2.24 h (90%)
WISP-CSE 150 75 75 62.44 / 58.74 (+3.70) 73.43 / 72.86 (+0.57) 62.94 / 59.10 (+3.84) 25.75 / 21.22 (+4.53) 2.58 h (88%)
WISP-ESE 150 70 80 60.77 / 57.38 (+3.39) 71.91 / 71.73 (+0.17) 61.01 / 57.71 (+3.30) 24.52 / 19.78 (+4.74) 3.93 h (89%)
WISP-Select5 150 112 38 69.27 / 67.10 (+2.17) 76.39 / 75.21 (+1.18) 69.81 / 66.94 (+2.87) 26.80 / 24.93 (+1.86) 0.64 h (31%)
WISP-Select7 150 112 38 68.86 / 66.69 (+2.17) 76.18 / 75.01 (+1.18) 69.52 / 66.65 (+2.87) 26.51 / 24.65 (+1.86) 0.65 h (29%)
WISP-Random (saved winner) 150 120 0 70.20 / 70.20 (+0.00) 77.28 / 77.28 (+0.00) 70.88 / 70.88 (+0.00) 28.25 / 28.25 (+0.00) 2.38 h (89%)
WISP-Evolution (saved winner) 150 120 0 69.59 / 69.59 (+0.00) 76.84 / 76.84 (+0.00) 70.17 / 70.17 (+0.00) 27.96 / 27.96 (+0.00) 2.06 h (89%)
WISP-Funnel, Random 150 116 4 69.20 / 69.22 (-0.02) 76.37 / 76.11 (+0.26) 70.18 / 70.01 (+0.16) 26.18 / 26.27 (-0.09) 6.26 h (78%)
WISP-Funnel, Evolution 150 119 5 69.76 / 69.30 (+0.46) 76.16 / 75.93 (+0.23) 70.10 / 69.97 (+0.13) 26.77 / 26.99 (-0.23) 7.14 h (85%)

"HMM kept": cells whose final model decodes with the HMM. "Decoder changed": cells where the switch chose another decoder than the recipe's own. Extra time: the decision (training-only fit and validation decoding; for WISPSelect only the validation prediction, since the selector keeps its training fits) plus any refit caused by a change of decoder, summed over the cells, and in brackets relative to the final fits' time.

Cells where the decoder changed: test change (adaptive minus paper, points)

Method Changed Macro-F1 up / down Mean Worst Best Balanced acc. mean / worst Accuracy mean / worst
WISP-SS 73 67 / 6 +8.03 -27.72 +16.22 +7.62 / -3.01 +1.68 / -7.43
WISP-RC 45 37 / 8 +12.62 -23.49 +27.12 +17.84 / -11.93 +7.98 / -22.79
WISP-SO 71 56 / 15 +6.51 -27.30 +15.86 +6.27 / -4.33 +1.25 / -9.41
WISP-GIS 74 58 / 16 +8.07 -24.85 +19.29 +6.67 / -8.94 +1.42 / -8.91
WISP-CIS 39 36 / 3 +16.18 -10.79 +25.38 +21.15 / -1.07 +9.84 / -14.49
WISP-CSE 75 66 / 9 +10.06 -23.16 +20.63 +9.74 / -5.79 +3.17 / -8.69
WISP-ESE 80 68 / 12 +8.96 -23.47 +22.26 +8.22 / -7.29 +2.44 / -9.64
WISP-Select5 38 36 / 2 +14.97 -23.49 +21.87 +18.64 / -11.93 +8.74 / -22.79
WISP-Select7 38 36 / 2 +14.97 -23.49 +21.87 +18.64 / -11.93 +8.74 / -22.79
WISP-Random (saved winner) 0
WISP-Evolution (saved winner) 0
WISP-Funnel, Random 4 3 / 1 -0.68 -23.91 +11.46 +5.46 / +0.21 +8.73 / +6.91
WISP-Funnel, Evolution 5 5 / 0 +12.54 +3.80 +25.30 +3.46 / -9.70 +6.22 / -9.07

HMM kept, by stream (of 15 cells; changed decoders in brackets)

Method ADL CAP24 DOMINO GOTOV HANDY HARMES-L HARMES-R IUWDS PAAL WISDM
WISP-SS 6 (9) 0 (15) 11 (4) 15 (0) 15 (0) 0 (15) 0 (15) 0 (15) 15 (0) 15 (0)
WISP-RC 9 (6) 14 (1) 13 (2) 15 (0) 15 (0) 0 (15) 0 (15) 13 (2) 11 (4) 15 (0)
WISP-SO 7 (8) 0 (15) 3 (12) 15 (0) 15 (0) 0 (15) 0 (15) 9 (6) 15 (0) 15 (0)
WISP-GIS 6 (9) 0 (15) 5 (10) 15 (0) 15 (0) 0 (15) 0 (15) 5 (10) 15 (0) 15 (0)
WISP-CIS 10 (5) 15 (0) 12 (3) 15 (0) 15 (0) 0 (15) 0 (15) 14 (1) 15 (0) 15 (0)
WISP-CSE 6 (9) 0 (15) 9 (6) 15 (0) 15 (0) 0 (15) 0 (15) 0 (15) 15 (0) 15 (0)
WISP-ESE 6 (9) 0 (15) 4 (11) 15 (0) 15 (0) 0 (15) 0 (15) 0 (15) 15 (0) 15 (0)
WISP-Select5 10 (5) 15 (0) 13 (2) 15 (0) 15 (0) 0 (15) 0 (15) 14 (1) 15 (0) 15 (0)
WISP-Select7 10 (5) 15 (0) 13 (2) 15 (0) 15 (0) 0 (15) 0 (15) 14 (1) 15 (0) 15 (0)
WISP-Random (saved winner) 15 (0) 15 (0) 15 (0) 15 (0) 15 (0) 0 (0) 0 (0) 15 (0) 15 (0) 15 (0)
WISP-Evolution (saved winner) 15 (0) 13 (0) 15 (0) 15 (0) 15 (0) 1 (0) 1 (0) 15 (0) 15 (0) 15 (0)
WISP-Funnel, Random 11 (1) 15 (2) 15 (0) 15 (1) 15 (0) 0 (0) 0 (0) 15 (0) 15 (0) 15 (0)
WISP-Funnel, Evolution 14 (1) 15 (2) 15 (0) 15 (2) 15 (0) 0 (0) 0 (0) 15 (0) 15 (0) 15 (0)

Per-cell rows: adaptive_decoder.csv.