Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / validation /software_validation_0.3.0.json
Zipeng365's picture
wisp-har 0.3.0: scikit-learn interface, recipe catalog, Optuna search, cost-accuracy figures
4731b11 verified
Raw History Blame Contribute Delete
4.16 kB
{
"version": "0.3.0",
"status": "passed",
"tested_commit_note": "Test runs used a wheel built from a snapshot of the 0.3.0 development tree, installed into a fresh virtual environment, with the tests run against the installed wheel (import paths checked by tests/test_installed.py). The commit after the final runs changed documentation, benchmark outputs and the requires-python metadata only.",
"environment": {
"python": "3.12.13",
"numpy": "2.3.5",
"scipy": "1.17.1",
"scikit-learn": "1.7.2",
"joblib": "1.5.3",
"threadpoolctl": "3.6.0",
"pytest": "9.0.3",
"blas_threads": 1
},
"runs": [
{"name": "full suite, C kernels compiled", "extras": "optuna, torch (CPU), skorch, pandas",
"equivalence_tests": true, "hub_test": "cached export", "slurm_job_id": "61805443",
"passed": 268, "failed": 0, "skipped": 0, "subtests_passed": 239},
{"name": "full suite, no C compiler (pure NumPy paths)", "extras": "optuna, torch (CPU), skorch, pandas",
"equivalence_tests": true, "hub_test": "cached export", "slurm_job_id": "61805444",
"passed": 254, "failed": 0, "skipped": 14, "skip_reasons": "14 tests of the optional native kernels",
"subtests_passed": 95},
{"name": "core install only (no optional extras), C kernels compiled", "equivalence_tests": false,
"slurm_job_id": "61805445", "passed": 203, "failed": 0, "skipped": 65,
"skip_reasons": "59 equivalence tests not requested; 3 torch, 2 optuna, 1 hub test skipped because the optional packages or data were absent",
"subtests_passed": 239},
{"name": "full suite on Python 3.11.14 (same pinned packages), C kernels compiled",
"equivalence_tests": true, "slurm_job_id": "61804925", "passed": 267, "failed": 0, "skipped": 1,
"skip_reasons": "hub test not requested", "subtests_passed": 239}
],
"equivalence": {
"E1_E2": "7 fixed families x ADL-Wrist fold 0 x seeds 42/43/44: decoded labels and class scores identical to wisp.registry.build_method; four metrics equal results/run_results.csv (largest difference 5.6e-17, from the CSV's decimal text)",
"E3": "14 CIS block-subset recipes on ADL-Wrist fold 0 seed 42 equal results/cis_and_budget_runs.csv",
"E4": "WISPRepresentation output bit-identical to the family-internal features for SS, RC, SO, GIS, CIS and 6 CIS subsets",
"E5": "4 search best recipes that differ from the fixed recipes, each refitted on a cell that selected it: with the candidate seed read from the retained checkpoint the saved metrics are reproduced exactly; with the catalog default seed the macro-F1 differs by -0.0054 to -0.0172",
"E6": "WISPSelect on ADL-Wrist fold 0 seeds 42/43/44 selects the saved winner (Select5: CIS x3; Select7: CSE x3) and reproduces the saved metrics",
"E7": "HMMSmoother identical to TransitionSmoother on real class scores (0/861 labels differ)",
"tables": "benchmarks/results/equivalence_adl_fold0_native.md, benchmarks/results/equivalence_adl_fold0_pure.md"
},
"retained_checkpoints": "The six ADL-Wrist fold-0 WISP-Random/WISP-Evolution checkpoints unpickle with the frozen module paths and reproduce their saved test metrics (largest difference 5.6e-17) with the 0.3.0 wheel installed; slurm jobs 61803737 (0.2.0 code at M0) and 61808334 (0.3.0)",
"quick_start": "README quick start run after downloading ADL-Wrist fold 0 from the Hub at the pinned revision: macro-F1 0.5805456914889277, equal to results/run_results.csv (wisp_cis, fold 0, seed 42); seeds 43/44 and WISP-RC also equal; slurm job 61804782",
"all_cell_check": "WISPClassifier refits of the catalog recipes on all 180 paper cells: speed=exact reproduces the saved four metrics in 3,599 of 3,600 cells (7 fixed families with HMM, CIS without HMM, 6 CIS subsets with and without HMM; largest difference 1.1e-16); one cell (CAPTURE-24 fold 2 seed 43 WISP-CSE) cannot be fitted in the pinned environment because RidgeClassifierCV raises LinAlgError (SVD did not converge), as does the paper code itself; speed=fast accuracy record over 900 cells in benchmarks/results/saved_results_check.md; slurm jobs 61805039, 61806253 and 61807732-61807746"
}