Instructions to use Zipeng365/WISP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Scikit-learn
How to use Zipeng365/WISP with Scikit-learn:
from huggingface_hub import hf_hub_download import joblib model = joblib.load( hf_hub_download("Zipeng365/WISP", "sklearn_model.joblib") ) # only load pickle files from sources you trust # read more about it here https://skops.readthedocs.io/en/stable/persistence.html - Notebooks
- Google Colab
- Kaggle
|
Download docs/RECIPES.md from Zipeng365/WISP: direct link, hf CLI and curl.
- Browser
- Download file 12.4 kB
-
https://huggingface.co/Zipeng365/WISP/resolve/main/docs/RECIPES.md
- Command line
-
hf download hf://Zipeng365/WISP/docs/RECIPES.md
-
curl -L -o RECIPES.md https://huggingface.co/Zipeng365/WISP/resolve/main/docs/RECIPES.md
12.4 kB
| # Which recipe, and when to search | |
| The catalog holds only recipes with rows in the saved result files (`results/`). Every | |
| number on this page comes from those files through `benchmarks/catalog_summary.py` | |
| (output: `benchmarks/results/catalog_summary.json`), from this package's own timing benchmark | |
| (`benchmarks/results/speed_engines.csv`, see [SPEED.md](SPEED.md)), or from the cost–accuracy | |
| figure data (`docs/figures/cost_accuracy_plotted_values.csv`); each statement says which. | |
| Paper averages below follow the paper's rule: mean over the 5 folds and 3 seeds (42, 43, 44) | |
| per view, then over the nine primary dataset views (the two HARMES wrist streams form one | |
| view; the nine-channel mHealth and PAMAP2 streams are not in this average). This rule | |
| reproduces the reported means exactly (e.g. WISP-CIS macro-F1 66.1391). They are test-split | |
| results of the paper protocol and are listed for orientation only: none of the advice on this | |
| page was chosen by looking at them (see "How to choose" below). | |
| ## How to choose | |
| 1. **Start with `"cis"` (WISP-CIS, the default).** It is the recipe that validation most often | |
| picks: WISP-Select5 chose CIS in 106 of its 180 validation selections (SS 44, RC 29, GIS 1; | |
| `models/model_inventory.csv` of the Hugging Face repository), and among the 360 final recipes | |
| of the budget-329 searches, each the validation winner of its search, the fixed WISP-CIS | |
| configuration is the most frequent one (53 of 360; `results/search_selections.csv`). It | |
| needs no search. | |
| 2. **Let validation choose among the fixed families with `WISPSelect(menu=5)`** when you have a | |
| validation split of separate participants. It fits five candidates on training data and | |
| refits the winner on train+validation. | |
| 3. **Try the search's best recipe of a similar data stream** (`wisp.recipes.best(...)`, table | |
| below), e.g. for similar sensors, window length and activity set. | |
| 4. **Search last, and try WISP-Funnel first** (`WISPSearch("funnel")`, [below](#wisp-funnel-a-fast-search)). | |
| One funnel run gives both a Random and an Evolution model for about 1/47 of the | |
| candidate-fitting time of one paper WISP-Random search, within one macro-F1 point of the | |
| full searches on the benchmark. The full searches (`WISPSearch("random" | "evolution")`) are | |
| expensive: on the ten primary streams the median cumulative candidate fitting time of one | |
| paper search (one dataset, fold and seed, 329 candidates) was 47.5 h for WISP-Random and | |
| 61.3 h for WISP-Evolution (`results/search_selections.csv`, budget 329, 150 searches each). | |
| The final model of a search is cheap; the cost is in the selection. | |
| Pass `groups` (and `time_index`) at fit and predict time when windows come in sequences: the | |
| default `decoder="adaptive"` then decides on validation data whether the HMM decoder helps | |
| (see [API.md](API.md#the-adaptive-hmm-switch-decoderadaptive)). In the paper, each recipe | |
| carried its own HMM flag; about four in five of the 360 final search recipes kept the HMM | |
| (Random 146/180, Evolution 144/180). The paper numbers on this page use each recipe's own | |
| decoder: reproduce them with `decoder="recipe"`. | |
| ## Fixed families | |
| | Recipe | Paper name | Macro-F1 | Balanced acc. | Accuracy | Worst-class F1 | | |
| |---|---|---:|---:|---:|---:| | |
| | `"ss"` | WISP-SS | 58.54 | 59.04 | 73.36 | 21.70 | | |
| | `"rc"` | WISP-RC | 63.44 | 63.59 | 71.46 | 22.87 | | |
| | `"so"` | WISP-SO | 53.59 | 54.19 | 69.31 | 16.75 | | |
| | `"gis"` | WISP-GIS | 55.16 | 55.95 | 70.66 | 19.49 | | |
| | `"cis"` | WISP-CIS | 66.14 | 66.04 | 74.07 | 24.78 | | |
| | `"cse"` | WISP-CSE | 58.74 | 59.10 | 72.86 | 21.22 | | |
| | `"ese"` | WISP-ESE | 57.38 | 57.71 | 71.73 | 19.78 | | |
| | `WISPSelect(5)` | WISP-Select5 | 67.10 | 66.94 | 75.21 | 24.93 | | |
| | `WISPSelect(7)` | WISP-Select7 | 66.69 | 66.65 | 75.01 | 24.65 | | |
| | `WISPSearch("random")` | WISP-Random | 70.20 | 70.88 | 77.28 | 28.25 | | |
| | `WISPSearch("evolution")` | WISP-Evolution | 69.59 | 70.17 | 76.84 | 27.96 | | |
| Source: `results/run_results.csv`, paper average (percent). All fixed families use the HMM decoder | |
| (`decoder="recipe"` reproduces these numbers). | |
| ## CIS block subsets | |
| `"cis:R"`, `"cis:I"`, `"cis:S"`, `"cis:RI"`, `"cis:RS"`, `"cis:IS"` (and `"cis:RIS"`, which is | |
| `"cis"`) keep only the named blocks of WISP-CIS: R random convolution, I interval summaries, | |
| S within-window state and spectral features. Each was run with and without the HMM decoder; | |
| switch with `decoder="hmm"` / `decoder="none"`. | |
| | Blocks | Macro-F1, HMM | Macro-F1, direct | | |
| |---|---:|---:| | |
| | R | 63.44 | 57.28 | | |
| | I | 43.89 | 38.63 | | |
| | S | 59.39 | 55.30 | | |
| | RI | 63.59 | 57.65 | | |
| | RS | 66.93 | 60.87 | | |
| | IS | 61.42 | 56.25 | | |
| | RIS (= `"cis"`) | 66.14 | 61.12 | | |
| Source: `results/cis_and_budget_runs.csv` (`kind == "recipe"`), paper average. These are a | |
| controlled block study, not a recommendation: they are test results, so do not pick a subset | |
| from this table for a new study; choose it on your own validation data (e.g. with | |
| `GridSearchCV` over `recipe`). | |
| ## Search best recipes | |
| Each search returns its validation winner. The best recipe of a data stream is the final | |
| recipe selected most often in its 30 searches (5 folds x 3 seeds x WISP-Random/WISP-Evolution), | |
| ties broken by the number of folds covered. Only validation decides; no test score is read. | |
| `scripts/build_search_best.py` computes the table from `results/search_selections.csv` and | |
| writes `src/wisp/_data/search_best.json`. | |
| | `wisp.recipes.best(...)` | Recipe | Selected | Folds covered | Same as | | |
| |---|---|---:|---:|---| | |
| | `"adl"` | RC, 256 kernels, Ridge, HMM | 9/30 | 3/5 | — (one of the search's initial templates) | | |
| | `"capture24"` | CIS, 128 kernels, 24 intervals, Ridge, HMM | 8/30 | 4/5 | — | | |
| | `"domino"` | CIS, 384 kernels, 24 intervals, Ridge, HMM | 11/30 | 4/5 | `"cis"` | | |
| | `"gotov"` | same as DOMINO | 20/30 | 5/5 | `"cis"` | | |
| | `"iuwds"` | same as DOMINO | 12/30 | 5/5 | `"cis"` | | |
| | `"harmes_left"` | CIS, 384 kernels, 24 intervals, Ridge, no HMM | 8/30 | 5/5 | `"cis"` with `decoder="none"` | | |
| | `"harmes_right"` | same as HARMES left | 7/30 | 4/5 | `"cis"` with `decoder="none"` | | |
| | `"handy"` | SO, HMM | 9/30 | 3/5 | `"so"` | | |
| | `"wisdm"` | GIS, ExtraTrees, 32 shapelets, HMM | 15/30 | 3/5 | `"gis"` | | |
| | `"paal_adl"` | SS, random forest, segments (5, 10, 20), 8 states, HMM | 8/30 | 4/5 | — (one of the search's initial templates) | | |
| | `"pamap2"` | RC, 256 kernels, Ridge, HMM | 6/30 | 3/5 | — (one of the search's initial templates) | | |
| | `"mhealth"` | SO, HMM | 6/30 | **1/5** | `"so"` | | |
| Notes: | |
| * Eight of the twelve are identical to a fixed recipe or to `"cis"` without the HMM, so they | |
| have results on every fold and seed. The other four (ADL, CAPTURE-24, PAAL-ADL, PAMAP2) have | |
| results only in the searches that selected them. | |
| * The mHealth best recipe comes from a single fold; its support is weak. | |
| * Across all 360 searches the most frequent final recipe is the fixed WISP-CIS (53 times), | |
| one reason it is the package default. | |
| * The recorded recipe does not include the candidate's random seed, and the search changes the | |
| seed with every mutation. Refitting a best recipe therefore gives a model with the same | |
| configuration but a different seed, whose metrics can differ slightly from the cells that | |
| selected it. Recipes identical to a fixed recipe are not affected: their default-seed | |
| results are in the result files. | |
| ## WISP-Funnel: a fast search | |
| `WISPSearch("funnel", controller="random" | "evolution")` runs both controllers of the paper's | |
| search in one pass on the CIS family: each controller's proposals are ranked on a | |
| training-only proxy (at most 4,096 training and 1,024 validation windows, a shared Ridge | |
| feature dictionary, cost weight 0), a shortlist of 4 per controller is verified on the full | |
| splits with the screened Ridge solver, and each winner is refitted on train+validation | |
| ([API.md](API.md#wisp-funnel-wispsearchstrategyfunnel)). It is this package's method and | |
| benchmark, not a paper result, and it is not the paper's search protocol. | |
| **Benchmark record.** 150 cells: the ten primary streams x 5 folds x 3 seeds (42, 43, 44), | |
| the paper's training and validation splits for selection, the test split scored once with each | |
| model's own per-person HMM decoder (`decoder="recipe"`). Averages follow the paper rule above | |
| (nine primary views). This package reproduces the cells of the record through | |
| `WISPSearch(strategy="funnel", decoder="recipe")` with each export's sampling rate and study | |
| setting passed to `fit` ([`benchmarks/results/funnel_record_check.md`](../benchmarks/results/funnel_record_check.md)). | |
| | Method | Macro-F1 | Accuracy | Balanced acc. | Worst-class F1 | | |
| |---|---:|---:|---:|---:| | |
| | WISP-Random (paper search) | 70.20 | 77.28 | 70.88 | 28.25 | | |
| | WISP-Evolution (paper search) | 69.59 | 76.84 | 70.17 | 27.96 | | |
| | **WISP-Funnel, Evolution controller** | **69.30** | 75.93 | 69.97 | 26.99 | | |
| | **WISP-Funnel, Random controller** | **69.22** | 76.11 | 70.01 | 26.27 | | |
| | WISP-CIS (fixed, no search) | 66.14 | 74.07 | 66.04 | 24.78 | | |
| The two funnel models are 0.98 (Random) and 0.29 (Evolution) macro-F1 points below the | |
| corresponding paper searches and 3.08 and 3.16 points above WISP-CIS. In paired exact | |
| sign-flip tests over the nine views (Holm-adjusted within each metric) none of these | |
| macro-F1 differences is significant (adjusted p >= 0.98). | |
| **Cost.** Per stream, the median over its 15 cells of the funnel's CPU time and the median | |
| over its 15 searches of one paper search's cumulative candidate-fitting time: | |
| | Stream | Funnel run, both controllers (core-h) | One WISP-Random search (core-h) | One WISP-Evolution search (core-h) | | |
| |---|---:|---:|---:| | |
| | ADL-Wrist | 0.022 | 4.04 | 5.37 | | |
| | CAPTURE-24 | 1.248 | 54.23 | 78.48 | | |
| | DOMINO | 0.242 | 20.02 | 33.31 | | |
| | GOTOV | 1.174 | 55.73 | 92.90 | | |
| | HANDY | 0.087 | 9.23 | 10.22 | | |
| | HARMES left wrist | 1.148 | 53.53 | 60.73 | | |
| | HARMES right wrist | 1.248 | 45.59 | 66.83 | | |
| | IUWDS | 1.281 | 61.14 | 112.44 | | |
| | PAAL-ADL | 0.144 | 11.17 | 15.56 | | |
| | WISDM | 1.163 | 50.89 | 65.60 | | |
| | **Sum of the ten medians** | **7.76** | **365.57** | **541.45** | | |
| So one funnel run, which yields both models, costs about 1/47 of one WISP-Random search and | |
| 1/70 of one WISP-Evolution search (sums of the per-stream medians; per stream 1/37 to 1/184 | |
| against WISP-Random). What is measured: for the funnel, the whole run's process CPU time (user | |
| plus system, both threads): both controllers' proxy search, verification and both refits on | |
| train+validation, on two cores of an Intel Xeon Platinum 8570. For the paper searches, the | |
| recorded cumulative single-core time of the 329 candidate fits of one search | |
| (`results/search_selections.csv`), which leaves out the search's validation and stress-view | |
| scoring and its final refit; the comparison therefore favours the paper searches. The largest | |
| funnel run took 1.07 h of wall time and at most 12.2 GiB of memory (both controllers in | |
| parallel). Source: the record's per-cell RESULT files, summarised in | |
| `benchmarks/results/funnel_record_check.md`. | |
| ## Cost notes | |
| * **Final models are cheap.** Fit on train+validation plus test prediction, core-hours over the | |
| ten primary streams (fold 0, seed 42): WISP-Random 0.311, WISP-CIS 0.347, WISP-Evolution | |
| 0.712 (`docs/figures/cost_accuracy_plotted_values.csv`, Xeon 8570, one thread). This package's | |
| own timings of every fixed family on all twelve streams are in | |
| `benchmarks/results/speed_engines.csv` (summary in [SPEED.md](SPEED.md)). | |
| * **Selection is the expensive part.** WISPSelect fits each candidate family once on training | |
| data before the refit; a search fits up to `budget` candidates and scores each on the | |
| validation split and four perturbed copies of it (median 47.5 h / 61.3 h per paper search, | |
| above). `WISPSelect`/`WISPSearch` report `selection_seconds_` and `final_fit_seconds_` | |
| separately. | |
| * **Speed tiers** change only the time, not the recipe: `speed="fast"` affects Ridge readouts | |
| (RC, CIS, the RC/CIS members of CSE and ESE, and RC/CIS search candidates) and can change | |
| results slightly; paper results correspond to `speed="exact"` only. | |
| * **Memory.** Peak host memory of one final model (train+validation fit plus test prediction, | |
| one thread, largest of the ten primary streams): WISP-Random 4.7, WISP-Evolution 8.0, WISP-CIS | |
| 8.9 GiB (`docs/figures/memory_plotted_values.csv`; per-stream values in | |
| `benchmarks/results/memory_final_models.csv`). | |