Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
WISP / FAILURES.md
Zipeng365's picture
wisp-har 0.4.0: WISP-Funnel search and the adaptive HMM switch (default decoder)
93a9282 verified
|
Raw History Blame Contribute Delete
11 kB
# Failure log (wisp-har development)
Every failed or aborted test run, benchmark job or submission during the 0.3.0 and 0.4.0 work is
recorded here with its cause, peak memory (when the job ran) and the fix. Job numbers are
Slurm job IDs of the development cluster.
| # | When (UTC date) | Job / step | What failed | Cause | Peak memory | Fix |
|---|---|---|---|---|---|---|
| 1 | 2026-11-14 | wheel test submission (M0) | `sbatch` rejected the test script twice | The script called `module load` without first resetting the module environment, which the scheduler refuses | not started | Removed the module line; the native build uses the system C compiler |
| 2 | 2026-11-14 | 61803296, 61803297 (M0 wheel tests) | Cancelled before running | An edit put a shell comment before `fi`, so the script had a syntax error | not started | Fixed the line, checked with `bash -n`, resubmitted as 61803340/61803341 (both passed) |
| 3 | 2026-11-14 | 61803906 (smoke run of the new layer) | Optuna trials proposing CIS block subsets failed (`TypeError: unexpected keyword argument 'use_intervals'`) | The search engine builds candidates with `wisp.cpu.algorithms`, whose CIS class has no block switches (they exist only in `wisp.cis_algorithms`, used by `wisp.ablations`) | 0.15 GB | Optuna candidates are scored with the frozen `evaluate_candidate_cascade` and an estimator factory that builds block subsets from the `wisp.cis_algorithms` template, wrapped by the accelerated engine as `wisp_fast.engine.build_recipe` does |
| 4 | 2026-11-14 | 61804028 (first wheel test of the new layer, native) | 3 of 207 tests failed | (a) the clone test compared lists of transformer objects by equality; (b) with `enable_metadata_routing=True` the kernel's internal `Pipeline.fit(..., clf__sample_weight=...)` is rejected by scikit-learn; (c) `check_estimator` reported checks that do not fit window input | 0.53 GB | (a) compare element types; (b) kernel calls run inside a thread-local `config_context(enable_metadata_routing=False)` (the routing of `groups` to the public estimators is unchanged); (c) see the exemption list in `tests/test_api_sklearn.py` |
| 5 | 2026-11-14 | 61804892 (Python 3.11 test run) | Cancelled seconds after submission | The `--export` list contained a version range with a comma (`>=3.11,<3.13`), which Slurm splits into two variables | not started | Resubmitted as 61804925 with `>=3.11` |
| 6 | 2026-11-14 | 61805039 task 7 (all-cell check, CAPTURE-24 fold 2) | `WISPClassifier("cse", random_state=43)` raised `LinAlgError: SVD did not converge` inside the RC member's `RidgeClassifierCV`, which stopped the task | The fixed-family kernel (`wisp.cpu.algorithms`) uses scikit-learn's `RidgeClassifierCV` without the gesvd retry that the search kernel (`wisp.cis_algorithms`) has; on this cell LAPACK's divide-and-conquer SVD did not converge with one BLAS thread. See the diagnosis entry below | 6.7 GB | The check script records such a cell as an error and continues (task resubmitted as 61806253); diagnosis in entry 7 |
| 7 | 2026-11-14 | 61806249, 61806250, 61806251, 61806270, 61806271 (diagnosis of entry 6) | The same cell (CAPTURE-24, fold 2, seed 43, WISP-CSE, train+validation fit) fails with `LinAlgError: SVD did not converge` in every configuration tried: the paper code itself (wisp-har 0.1.0, `wisp.registry.build_method`) with 1 and 8 BLAS threads, and `WISPClassifier` with 8 threads and with 1 thread on Ice Lake, Sapphire Rapids and Emerald Rapids nodes | The failure is in scikit-learn's `RidgeClassifierCV` (SciPy `gesdd`) on the RC member's features under the pinned NumPy 2.3.5 / SciPy 1.17.1 / scikit-learn 1.7.2; it is not caused by the 0.3.0 layer or the accelerated engine (the paper code fails identically). The saved result for this cell was produced in an environment where the SVD converged | 2.8–6.7 GB | Not patched (the kernel is frozen). Documented as a known limitation in docs/SPEED.md and docs/API.md; the all-cell check reports the cell as not reproducible in this environment |
| 8 | 2026-11-15 | 61804804 (Optuna-vs-Random benchmark), tasks 1, 3, 5, 49, 51, 53 | Slurm accounting still listed these six tasks as RUNNING (elapsed frozen at 1-00:11:53, end "Unknown") although `squeue` no longer showed them | Their logs show each finished on 2026-11-14 and wrote its result JSON and peak memory (largest 1.16 GiB, MaxRSS); the stale state is an accounting artefact, not a failed run. All 72 tasks produced results | no rerun needed; tables generated from the 72 result files |
| 9 | 2026-11-15 | 61856317 task 16 (final-model memory, WISP-Evolution, HARMES right wrist, fold 0, seed 42) | `RidgeClassifierCV` raised `LinAlgError: SVD did not converge` with one BLAS thread (the selected recipe: RC, 768 kernels, Ridge, no HMM; peak before the error 5.47 GiB) | Same LAPACK non-convergence as entries 6–7 | Rerun as 61856627 with `--blas-threads 4`: converged, the four metrics equal `results/run_results.csv` (largest difference 5.6e-17), peak 5.59 GiB; recorded in `benchmarks/results/memory_final_models.csv` (column `blas_threads`) |
## 0.4.0 development
| # | When (UTC date) | Job / step | What failed | Cause | Peak memory | Fix |
|---|---|---|---|---|---|---|
| 10 | 2026-11-16 | 61882274 (first full test run of the 0.4.0 working tree) | Cancelled seconds after submission, before pytest started | It was submitted with `pytest -x`, which stops at the first failure and would hide later ones | not started | Resubmitted without `-x` as 61882275 |
| 11 | 2026-11-16 | 61882275 (full test suite with the equivalence checks, native kernels) | 1 of 288 tests failed: `test_search_scores_candidates_direct_without_sequences[random]` (287 passed, 1 skipped: the Hub download test) | The new test compared the validation utilities of two `WISPSearch("random")` runs; the utility contains a fit-time cost term (`cost_weight=0.01`), so two runs differ in the 6th digit (0.97689737 vs 0.97689415). The decoders and candidate scores themselves were as intended | 8.1 GiB (MaxRSS of pytest) | The test sets `cost_weight=0` for the Random strategy and also compares the validation macro-F1 of every candidate; the equivalence report of this run (E1–E7, 59 records) is identical to the 0.3.0 report apart from timings |
| 12 | 2026-11-16 | 61882595, tasks 0–3 (sampled check of `speed="exact"`/`"fast"` against the saved paper cells) | All four tasks stopped after 5–7 s: `can't open file .../check_against_saved_results.py` | The job script pointed at a `src_tree/` subfolder, but the venv builder unpacks the commit directly into the build folder | 8 MB | Path corrected; resubmitted as 61883165 |
| 13 | 2026-11-16 | 61882594, the 150 `search` tasks of the `decoder="adaptive"` measurement | Every WISP-Random row ended with `ValueError: could not convert string to float: ''` (WISP-Evolution rows were fine) | The measurement script read the paper's `fit_seconds` from `results/run_results.csv` to report it next to the new fit time; that column is empty for all 180 `wisp_random` rows. The error came after the Random model had been fitted and scored, so only the bookkeeping failed; the package and the public `benchmarks/adaptive_decoder.py` (which does not read the column) are unaffected | 14.0 GiB (largest task of the array) | An empty `fit_seconds` is now read as missing. The 150 old `search` outputs were archived and the tasks rerun (61884519, and a second round for the 21 tasks still running at the time of the fix) |
| 14 | 2026-11-16 | 61882594 tasks of the `decoder="adaptive"` measurement: WISP-RC and WISP-Select5/Select7 on WISDM fold 1 seed 44, WISP-CSE on CAPTURE-24 fold 2 seed 43, WISP-Evolution's saved recipe on HARMES right fold 0 seed 42 | `LinAlgError: SVD did not converge` in scikit-learn's `RidgeClassifierCV` with one BLAS thread | The LAPACK non-convergence of entries 6–7 and 9. The adaptive switch fits the selected recipe once more on the training split alone (the decision fit); on WISDM fold 1 seed 44 that training-only RC fit does not converge with one thread, while the paper's train+validation fit does | see the retries | Retried those methods alone with 4 BLAS threads (61884684, 61884685, 61884686; HARMES right after the search rerun); outcomes in the next entry |
| 15 | 2026-11-16 | Outcome of the retries of entry 14 (61884684, 61884685, 61884686, 61885046; 4 BLAS threads) | β€” | β€” | 4.7, 3.1, 17.0 and 5.8 GiB | All converged. WISP-RC (WISDM fold 1 seed 44) kept the HMM and WISP-Select5/Select7 kept their HMM winners (CIS, ESE): all four metrics equal `results/run_results.csv` (largest difference 9.7e-17). WISP-Evolution's saved recipe on HARMES right fold 0 seed 42 (no HMM) kept argmax and equals the saved result (5.6e-17). WISP-CSE on CAPTURE-24 fold 2 seed 43 (the cell of entries 6–7, which the paper code cannot fit in this environment) fitted with 4 threads; the switch chose argmax, so no saved result exists for that configuration. These rows are marked `blas_threads = 4` in the measurement output |
| 16 | 2026-11-16 | 61882592 (WISP-Funnel through the public API, `decoder="recipe"`, 150 record cells) | 8 of the 10 streams reproduced the record bit for bit; on CAPTURE-24 and PAAL-ADL 46 of the 60 controller runs chose a different winner (the Random controller often reached the same recipe under another candidate ID) | The search controllers' dataset fingerprint reads the sampling rate and the metadata `study_setting` (a setting containing "free", "home", "natural" or "wild" sets its free-living motif weight). The record's data objects carried both from the WristHARBench export; `WISPSearch` built its data objects without them, so the two free-living streams lost the flag. The same gap exists in 0.3.0's `WISPSearch("random" \| "evolution")` | 12.2 GiB (largest cell) | `WISPSearch.fit(..., sampling_rate_hz=..., study_setting=...)` (defaults keep the old behaviour) and `wisp.datasets` folds expose both values. 61882592 was cancelled with 3 cells left and the adaptive run 61882593 at 122 of 150; both were rerun with the descriptors (61885546, 61885547) |
| 17 | 2026-11-16 | 61804804 (0.3.0 Optuna-vs-Random benchmark), the 6 WISP-Random arms on CAPTURE-24 and PAAL-ADL (fold 0, seeds 42–44) | Not a crash: these arms ran without the free-living flag of entry 16, so they were not the paper's WISP-Random configuration as the table states | Entry 16. The Optuna arms do not use a dataset fingerprint and are unaffected | 15.4 GiB (largest task of 61804804) | The 6 arms were rerun with the descriptors (61885780; same budget 329, seeds, `n_jobs=2`, one BLAS thread); `benchmarks/results/optuna_vs_random.*` regenerated, the superseded rows kept in `benchmarks/results/optuna_vs_random_superseded.csv`. All six reruns completed (CAPTURE-24 9.3–9.7 h, PAAL-ADL 2.0 h; peak 9.8 GiB) and chose the same recipes; five have the same test result, PAAL-ADL seed 44 changed from 0.7906 to 0.7407 macro-F1 (another candidate seed); mean Optuna minus Random test macro-F1 +0.27 points (was +0.13) |