Scikit-learn
human-activity-recognition
wearable
wrist
time-series
cpu
scikit-learn
Zipeng365 commited on
Commit
103792b
·
verified ·
1 Parent(s): 19383e1

README: Pareto optimality and dominance counts for the cost and memory figures

Browse files
Files changed (2) hide show
  1. README.md +34 -16
  2. SHA256SUMS +1 -1
README.md CHANGED
@@ -23,9 +23,22 @@ WISP (Wrist-Informed Space of Predictive Representations) is a CPU method family
23
  ## Why WISP
24
 
25
  WISP is accurate, fast and small, and it plugs into the tools you already use. Each figure below
26
- places every method by one score against one resource; the dashed gold line is the WISP Pareto
27
- frontier, and the further up and to the left a point sits, the better. WISP points use the
28
- paper protocol (`speed="exact"`).
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
  ### 1. Accurate and fast
31
 
@@ -46,11 +59,13 @@ selection at all. One NVIDIA A100 hour counts as 64 core-hours. Every plotted va
46
  0.712 for WISP-Evolution. The final models of tuned RDST, MR-Hydra, MultiRocket and
47
  MiniRocket cost 34.15, 45.52, 84.63 and 15.53 core-hours, 50–272 times WISP-Random's. WISP
48
  runs on CPUs only; no GPU is needed.
49
- - **Fair comparison.** One baseline is cheaper: Balanced-RF's final model (0.22 core-hours,
50
- 47.85 macro-F1) costs less than every WISP point. Every other baseline has a WISP point that is
51
- both cheaper and more accurate on all four metrics. The axis leaves out configuration
52
- selection, where a WISP search is expensive ([cost notes](docs/RECIPES.md#cost-notes)); fixed
53
- WISP-CIS has no selection cost.
 
 
54
 
55
  ### 2. Good on every class, at a fraction of the time
56
 
@@ -61,9 +76,10 @@ ones) against the same final-model cost axis.*
61
 
62
  Wearable activity data are imbalanced: a few activities fill most of the day. Balanced accuracy
63
  rewards a model only if it recognises the rare activities too. WISP-Random (70.88) and
64
- WISP-Evolution (70.17) are the two most balanced methods in the comparison; the best baseline,
65
- tuned RDST, reaches 68.21 with a final model that costs 110 times WISP-Random's (34.15 against
66
- 0.311 core-hours).
 
67
 
68
  ### 3. Small memory
69
 
@@ -82,11 +98,13 @@ baselines (their final refits were recorded with 1–8 threads, which is not com
82
  these single-thread peaks), HC2 (32 threads) and the deep models (GPU memory). Values:
83
  [`docs/figures/memory_plotted_values.csv`](docs/figures/memory_plotted_values.csv).*
84
 
85
- The final model of WISP-Random peaks at 4.7 GiB, WISP-Evolution at 8.0 GiB and WISP-CIS at
86
- 8.9 GiB, so all three fit on a laptop with 16 GB of memory. RDST, MR-Hydra and MultiRocket needed
87
- 104–114 GiB, about 22–24 times WISP-Random, and MiniRocket 21.8 GiB, while scoring 10–11
88
- points lower in macro-F1. Two baselines use less memory than WISP-Random: QUANT (4.4 GiB) and Balanced-RF
89
- (2.3 GiB), at 14.9 and 22.3 points lower macro-F1.
 
 
90
 
91
  ### 4. Extensible
92
 
 
23
  ## Why WISP
24
 
25
  WISP is accurate, fast and small, and it plugs into the tools you already use. Each figure below
26
+ places every method by one score against one resource, so the further up and to the left a
27
+ point sits, the better. A method is *Pareto-dominated* when another method needs no more of the
28
+ resource and scores higher; every point beneath the dashed gold line (the WISP Pareto frontier)
29
+ is dominated by a WISP method. WISP points use the paper protocol (`speed="exact"`).
30
+
31
+ **WISP-Random is Pareto-optimal in all eight figures** (four metrics, each against final-model
32
+ cost and against memory): no method is both cheaper, or smaller, and more accurate. It
33
+ dominates most baselines outright:
34
+
35
+ | Resource (same result for all four metrics) | Overall Pareto frontier | Baselines Pareto-dominated by WISP |
36
+ |---|---|---|
37
+ | Final-model cost | WISP-Random, Balanced-RF | **14 of 15** by WISP-Random and by WISP-Evolution; 11 by WISP-CIS, which needs no search |
38
+ | Peak memory | WISP-Random, QUANT, Balanced-RF | **4 of 6** by each of WISP-Random, WISP-Evolution and WISP-CIS |
39
+
40
+ The baselines that stay on the frontier do so only by being cheaper or smaller, not by being
41
+ more accurate: Balanced-RF scores 22.3 macro-F1 points below WISP-Random, and QUANT 14.9.
42
 
43
  ### 1. Accurate and fast
44
 
 
59
  0.712 for WISP-Evolution. The final models of tuned RDST, MR-Hydra, MultiRocket and
60
  MiniRocket cost 34.15, 45.52, 84.63 and 15.53 core-hours, 50–272 times WISP-Random's. WISP
61
  runs on CPUs only; no GPU is needed.
62
+ - **Pareto-dominant.** WISP-Random and WISP-Evolution are both cheaper and more accurate than
63
+ 14 of the 15 baselines, including every tuned baseline, HC2 and all four deep models. Fixed
64
+ WISP-CIS, with no search at all, dominates 11 of them; only Balanced-RF and the three strongest
65
+ tuned baselines (RDST, MR-Hydra, MultiRocket) escape it. Balanced-RF's final model (0.22
66
+ core-hours, 47.85 macro-F1) is the only one cheaper than every WISP point.
67
+ - **What the axis leaves out.** Configuration selection is not on it; a WISP search is expensive
68
+ there ([cost notes](docs/RECIPES.md#cost-notes)), while fixed WISP-CIS has no selection cost.
69
 
70
  ### 2. Good on every class, at a fraction of the time
71
 
 
76
 
77
  Wearable activity data are imbalanced: a few activities fill most of the day. Balanced accuracy
78
  rewards a model only if it recognises the rare activities too. WISP-Random (70.88) and
79
+ WISP-Evolution (70.17) are the two most balanced methods in the comparison, and both
80
+ Pareto-dominate 14 of the 15 baselines on this figure too. The best baseline, tuned RDST,
81
+ reaches 68.21 with a final model that costs 110 times WISP-Random's (34.15 against 0.311
82
+ core-hours).
83
 
84
  ### 3. Small memory
85
 
 
98
  these single-thread peaks), HC2 (32 threads) and the deep models (GPU memory). Values:
99
  [`docs/figures/memory_plotted_values.csv`](docs/figures/memory_plotted_values.csv).*
100
 
101
+ WISP-Random is again Pareto-optimal, and every WISP method Pareto-dominates RDST, MR-Hydra,
102
+ MultiRocket and MiniRocket on all four metrics. The final model of WISP-Random peaks at
103
+ 4.7 GiB, WISP-Evolution at 8.0 GiB and WISP-CIS at 8.9 GiB, so all three fit on a laptop with
104
+ 16 GB of memory. RDST, MR-Hydra and MultiRocket needed 104–114 GiB, about 22–24 times
105
+ WISP-Random, and MiniRocket 21.8 GiB, while scoring 10–11 points lower in macro-F1. QUANT
106
+ (4.4 GiB) and Balanced-RF (2.3 GiB) use less memory than WISP-Random, at 14.9 and 22.3 points
107
+ lower macro-F1.
108
 
109
  ### 4. Extensible
110
 
SHA256SUMS CHANGED
@@ -4,7 +4,7 @@
4
  0d9a22c8ec8957c6f775ff51432b7a33ca7dc9d68810f7d5553aa6ae82016239 CITATION.bib
5
  a553ad9270a356f4610a5c0c837ed7393ba2e0959c1c36620b62f9a299796698 FAILURES.md
6
  59e1ab474672b657909afcb346d7c6b2fc0fc627f6518b99925779aa1892b6c4 LICENSE
7
- 8a8304538102b4ad861253858b909fff8773cf33315abd27cf4f6b34f0942cb5 README.md
8
  43f14bd6fe7e2a93fc525a26fd95ea9cce48cd11b8ad927f1d4e0fbd2147c01c RELEASE_SUMMARY.json
9
  a621333daaa3a4d473eb30e9a1c13b3e8ec63e0fe0e21b0545ad64a7498bc2d1 SOURCE_PROVENANCE.json
10
  75489085ceb877e9cbdf5ba0fd1aeae3e7da5d11982f8c8c72482770e4a1a225 benchmarks/README.md
 
4
  0d9a22c8ec8957c6f775ff51432b7a33ca7dc9d68810f7d5553aa6ae82016239 CITATION.bib
5
  a553ad9270a356f4610a5c0c837ed7393ba2e0959c1c36620b62f9a299796698 FAILURES.md
6
  59e1ab474672b657909afcb346d7c6b2fc0fc627f6518b99925779aa1892b6c4 LICENSE
7
+ bf5b6960043b74b2458ccc68515130e0e4c28beb0ec1a2d2653ef500a698cc2c README.md
8
  43f14bd6fe7e2a93fc525a26fd95ea9cce48cd11b8ad927f1d4e0fbd2147c01c RELEASE_SUMMARY.json
9
  a621333daaa3a4d473eb30e9a1c13b3e8ec63e0fe0e21b0545ad64a7498bc2d1 SOURCE_PROVENANCE.json
10
  75489085ceb877e9cbdf5ba0fd1aeae3e7da5d11982f8c8c72482770e4a1a225 benchmarks/README.md