# Benchmarks and checks of this package Everything in this folder is **this package's own benchmark or check output, not a paper result**, except where a file compares with the saved paper results in `results/` (and then it says so). Inputs are frozen WristHARBench fold exports (`//fold/{train,valid,test}.npz`, as written by `scripts/export_benchmark.py` or `wisp.datasets.load_wristharbench`). | Script | What it does | Output in `results/` | |---|---|---| | `speed_engines.py`, `summarize_speed.py` | Paper code against `speed="exact"` and `speed="fast"` on the same cells: identity of class scores and labels, fit+predict time (one thread, one CPU type) | `speed_engines.csv`, `speed_engines.md` (used by docs/SPEED.md) | | `check_against_saved_results.py`, `summarize_saved_check.py` | `WISPClassifier` refits of the catalog recipes on all 180 paper cells (12 streams x 5 folds x 3 seeds), compared with `results/run_results.csv` and `results/cis_and_budget_runs.csv`; `speed="fast"` gives the fast tier's accuracy record | `saved_results_check.csv`, `saved_results_check.md` | | `check_checkpoints.py` | Loads the retained ADL fold-0 WISP-Random/WISP-Evolution checkpoints and rescores them against `results/run_results.csv` | (reported in the 0.3.0 validation record) | | `catalog_summary.py` | Paper averages and search costs computed from `results/` (no fitting) | `catalog_summary.json` (used by docs/RECIPES.md) | | `summarize_equivalence.py` | Tables of the equivalence tests E1–E7 (`tests/test_equivalence.py`) | `equivalence_adl_fold0_native.md`, `equivalence_adl_fold0_pure.md` | | `optuna_vs_random.py`, `summarize_optuna.py` | `WISPSearch("optuna")` (TPE) against `WISPSearch("random")` at equal budget (329 candidates), 12 streams x fold 0 x seeds 42/43/44. Selection uses validation only; the refitted winner is scored once on test for reporting. In 0.4.0 the WISP-Random arms on CAPTURE-24 and PAAL-ADL were rerun with the export's sampling rate and study setting (see CHANGELOG) | `optuna_vs_random.csv`, `optuna_vs_random.md`; superseded 0.3.0 rows: `optuna_vs_random_superseded.csv` | | `funnel_cells.py` | WISP-Funnel (`WISPSearch(strategy="funnel")`) on one cell through the public API, both controllers from one run; `--decoder recipe` is the protocol of the funnel benchmark record (150 cells: ten primary streams x 5 folds x 3 seeds, emerald nodes, 2 cores, one BLAS thread) | `funnel_record_check.md` (reproduction of the record, 9-view summary, costs) | | `adaptive_decoder.py` | `decoder="adaptive"` on one cell for the seven fixed families, WISPSelect and the saved WISP-Random/WISP-Evolution winners, with the paper's validation split as decision data; compared with `results/run_results.csv` | `adaptive_decoder.csv`, `adaptive_decoder.md` | | `memory_final_models.py` | Peak host memory (`ru_maxrss`) of one final model, one thread: WISP-Random and WISP-Evolution (the recipe each search selected at budget 329, refitted with the candidate seed of the retained checkpoint) and WISP-CIS, on the ten primary streams, fold 0, seed 42; each refit's four metrics are compared with `results/run_results.csv` (all equal) | `memory_final_models.csv` (per stream; the figure values are in `docs/figures/memory_plotted_values.csv`) | Notes: * Timing depends on the CPU, thread count and load; the speed tables were measured with one thread on one CPU type and the engines of each cell ran on the same node. * The search utility contains a measured-time term, so a search's winner can differ between machines; the Optuna-vs-Random numbers are one run per cell. * `_frozen.py` reads the fold exports (numeric arrays only, `allow_pickle=False`).