Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / benchmarks /README.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
3.7 kB

Benchmarks and checks of this package

Everything in this folder is this package's own benchmark or check output, not a paper result, except where a file compares with the saved paper results in results/ (and then it says so). Inputs are frozen WristHARBench fold exports (<root>/<dataset_id>/fold<k>/{train,valid,test}.npz, as written by scripts/export_benchmark.py or wisp.datasets.load_wristharbench).

Script What it does Output in results/
speed_engines.py, summarize_speed.py Paper code against speed="exact" and speed="fast" on the same cells: identity of class scores and labels, fit+predict time (one thread, one CPU type) speed_engines.csv, speed_engines.md (used by docs/SPEED.md)
check_against_saved_results.py, summarize_saved_check.py WISPClassifier refits of the catalog recipes on all 180 paper cells (12 streams x 5 folds x 3 seeds), compared with results/run_results.csv and results/cis_and_budget_runs.csv; speed="fast" gives the fast tier's accuracy record saved_results_check.csv, saved_results_check.md
check_checkpoints.py Loads the retained ADL fold-0 WISP-Random/WISP-Evolution checkpoints and rescores them against results/run_results.csv (reported in the 0.3.0 validation record)
catalog_summary.py Paper averages and search costs computed from results/ (no fitting) catalog_summary.json (used by docs/RECIPES.md)
summarize_equivalence.py Tables of the equivalence tests E1–E7 (tests/test_equivalence.py) equivalence_adl_fold0_native.md, equivalence_adl_fold0_pure.md
optuna_vs_random.py, summarize_optuna.py WISPSearch("optuna") (TPE) against WISPSearch("random") at equal budget (329 candidates), 12 streams x fold 0 x seeds 42/43/44. Selection uses validation only; the refitted winner is scored once on test for reporting. In 0.4.0 the WISP-Random arms on CAPTURE-24 and PAAL-ADL were rerun with the export's sampling rate and study setting (see CHANGELOG) optuna_vs_random.csv, optuna_vs_random.md; superseded 0.3.0 rows: optuna_vs_random_superseded.csv
funnel_cells.py WISP-Funnel (WISPSearch(strategy="funnel")) on one cell through the public API, both controllers from one run; --decoder recipe is the protocol of the funnel benchmark record (150 cells: ten primary streams x 5 folds x 3 seeds, emerald nodes, 2 cores, one BLAS thread) funnel_record_check.md (reproduction of the record, 9-view summary, costs)
adaptive_decoder.py decoder="adaptive" on one cell for the seven fixed families, WISPSelect and the saved WISP-Random/WISP-Evolution winners, with the paper's validation split as decision data; compared with results/run_results.csv adaptive_decoder.csv, adaptive_decoder.md
memory_final_models.py Peak host memory (ru_maxrss) of one final model, one thread: WISP-Random and WISP-Evolution (the recipe each search selected at budget 329, refitted with the candidate seed of the retained checkpoint) and WISP-CIS, on the ten primary streams, fold 0, seed 42; each refit's four metrics are compared with results/run_results.csv (all equal) memory_final_models.csv (per stream; the figure values are in docs/figures/memory_plotted_values.csv)

Notes:

  • Timing depends on the CPU, thread count and load; the speed tables were measured with one thread on one CPU type and the engines of each cell ran on the same node.
  • The search utility contains a measured-time term, so a search's winner can differ between machines; the Optuna-vs-Random numbers are one run per cell.
  • _frozen.py reads the fold exports (numeric arrays only, allow_pickle=False).