Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / docs /SPEED.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
7.71 kB

Speed tiers: speed="exact" and speed="fast"

wisp-har exposes exactly two speed tiers. Both run the frozen paper algorithms; they differ only in how the work is executed.

Parameter Promise Implementation chosen
speed="exact" (default) Outputs identical to the paper code The 0.2.0 accelerated engine (wisp_fast) with the original Ridge solver
speed="fast" Fastest; results can change slightly The same engine with the screened Ridge solver (Ridge strength chosen on a 512-row training subsample, then one Ridge fit on all rows). Only Ridge readouts change: WISP-RC, WISP-CIS and its subsets, the RC/CIS members of WISP-CSE and WISP-ESE, RC/CIS search candidates

Statements about paper results apply to speed="exact" only. The optional C kernels change nothing but time: without a C compiler the engine runs pure NumPy paths with the same outputs (the full test suite, including the equivalence tests below, passes both ways).

All numbers on this page are this package's own benchmark output, not paper results.

speed applies to the fixed recipes, WISPSelect and the Random, Evolution and Optuna searches. WISP-Funnel (WISPSearch(strategy="funnel")) has one protocol and its own engine (below); speed does not change it.

The exact tier: identical to the paper code, 5.7 times faster

speed="exact" was run next to the paper code (wisp-har 0.1.0, wisp.registry.build_method) on the same cells:

  • Cells: all seven fixed families on all twelve data streams, fold 0, seed 42 (84 cells), plus ADL-Wrist fold 0 seeds 43 and 44 (14 cells): 98 cells.
  • Protocol: the paper's fixed-family runner: fit on train+validation with participant groups and time index, predict the test split with the family's HMM decoder; per-window class scores saved for comparison.
  • Environment: Python 3.12.13 with the pinned NumPy 2.3.5, SciPy 1.17.1, scikit-learn 1.7.2; one thread (BLAS and n_jobs = 1); every run on the same CPU type (Intel Xeon Platinum 8360Y, 2.40 GHz); the paper code and the package ran one after another on the same node for each cell.
  • Script: benchmarks/speed_engines.py, summarised by benchmarks/summarize_speed.py into benchmarks/results/speed_engines.csv and .md.

Identity. On all 98 cells speed="exact" reproduces the paper code bit for bit: 0 decoded-label mismatches, 0 per-window argmax mismatches, maximum absolute class-score difference 0.

Time (fit + predict, seconds summed over the 12 streams, fold 0, seed 42):

Family Paper code speed="exact" Speed-up
WISP-SS 934 462 2.02x
WISP-RC 5,458 253 21.59x
WISP-SO 655 436 1.50x
WISP-GIS 2,623 584 4.49x
WISP-CIS 6,845 955 7.17x
WISP-CSE 4,850 904 5.37x
WISP-ESE 7,576 1,515 5.00x
All 84 cells 28,942 5,108 5.67x

The fast tier: screened Ridge

speed="fast" keeps everything of the exact tier except the Ridge regularisation search: the Ridge strength is chosen on a 512-row subsample of the training data, then one Ridge fit runs on all rows. This is the --speed fast option of 0.2.0. Time on the same cells (Ridge families, fold 0, seed 42, same node type and protocol):

Family Paper code speed="exact" speed="fast"
WISP-RC, 12 streams 5,458 s 253 s 103 s
WISP-CIS, 12 streams 6,845 s 955 s 202 s

The fast tier was chosen for its time and for having an accuracy record over all paper cells (next section), not for scoring better on test data.

Accuracy record of the fast tier and all-cell check

Every catalog recipe with a fixed configuration was refitted with WISPClassifier on all 180 paper cells (12 streams x 5 folds x seeds 42/43/44, train+validation fit, test prediction, one thread) and compared with the saved results (benchmarks/check_against_saved_results.py; outputs benchmarks/results/saved_results_check.csv and .md; Slurm jobs 61805039 and 61806253, plus 15 helper jobs for the last WISDM cells).

speed="exact": 3,599 of the 3,600 cells reproduce the saved four metrics exactly (largest difference 1.1e-16, the rounding of the CSV's decimal text): the seven fixed families with the HMM (1,260 cells), WISP-CIS without the HMM (180) and the six CIS block subsets with and without the HMM (2,160). The remaining cell, CAPTURE-24 fold 2 seed 43 WISP-CSE, cannot be fitted in the pinned environment: scikit-learn's RidgeClassifierCV stops with LinAlgError: SVD did not converge in the RC member. The paper code itself (wisp-har 0.1.0) fails identically with 1 and 8 BLAS threads, and so does WISPClassifier on three CPU generations (FAILURES.md, entries 6–7); the saved result was presumably produced in an environment where the SVD converged. With speed="fast" the same cell fits and gives the saved metrics.

speed="fast": change of the paper average macro-F1 (nine views, all folds and seeds):

Recipe Exact Fast Change
"cis" (HMM) 66.14 65.85 βˆ’0.28
"cis", decoder="none" 61.12 61.04 βˆ’0.08
"rc" 63.44 62.26 βˆ’1.19
"cse" 58.74 58.56 βˆ’0.18
"ese" 57.38 57.35 βˆ’0.03

SS, SO and GIS have no Ridge readout and are identical in both tiers (the engine builds them through the same code path). Mean macro-F1 change per data stream (15 cells each; range over cells in brackets):

Stream "cis" "rc"
ADL-Wrist +0.00 (+0.00 to +0.00) βˆ’0.32 (βˆ’2.17 to +0.00)
CAPTURE-24 +0.00 (+0.00 to +0.00) +0.00 (+0.00 to +0.00)
DOMINO +0.00 (+0.00 to +0.00) βˆ’1.07 (βˆ’6.96 to +0.85)
GOTOV βˆ’1.72 (βˆ’3.43 to βˆ’0.28) βˆ’3.35 (βˆ’7.47 to +1.86)
HANDY βˆ’0.35 (βˆ’1.46 to +0.36) βˆ’1.04 (βˆ’2.56 to +0.26)
HARMES left wrist +0.00 (+0.00 to +0.00) +0.00 (+0.00 to +0.00)
HARMES right wrist +0.00 (+0.00 to +0.00) βˆ’2.87 (βˆ’3.57 to βˆ’2.13)
IUWDS +0.24 (βˆ’1.62 to +4.54) βˆ’1.13 (βˆ’5.85 to +2.10)
mHealth +0.68 (βˆ’4.60 to +5.09) +0.39 (βˆ’0.97 to +6.49)
PAAL-ADL +0.00 (+0.00 to +0.00) βˆ’1.84 (βˆ’12.62 to +0.00)
PAMAP2 +0.00 (βˆ’0.00 to +0.00) βˆ’1.50 (βˆ’6.35 to +0.61)
WISDM βˆ’0.73 (βˆ’2.81 to +0.42) βˆ’0.50 (βˆ’3.93 to +1.56)

Where the screened subsample picks the same Ridge strength as the full leave-one-out search, the results are identical; elsewhere they change in both directions, by up to several points on single cells. The per-stream tables for CSE, ESE and direct CIS are in benchmarks/results/saved_results_check.md. The 0.2.0 release notes quoted per-dataset changes for other cells (GOTOV βˆ’3.44, HANDY +3.12 macro-F1 for WISP-CIS); the table above supersedes them as the fast tier's record over all 180 cells.

The WISP-Funnel engine

WISP-Funnel runs on a separate engine, wisp_fast.funnel, that shares no module with the engine above, so neither changes the other's results. Its verification and refit use the screened Ridge solver as part of the funnel's protocol. It has four optional C kernels (wisp_fast.funnel._kernels, _family_kernels, _cis_intervals, _cis_lite), compiled with the same flags as the kernels above. On two benchmark cells (ADL-Wrist fold 1 seed 42 and fold 3 seed 44) a build without a C compiler gave the same winners, test class scores and labels as the compiled build and the benchmark record, in 1.6 to 1.7 times the time (31 s against 18 s, and 56 s against 35 s, for the whole joint run). The funnel's accuracy and cost record is in RECIPES.md.