| # Scorecard — Apple arm64 (MPS) |
|
|
| 400 steps, batch 4, seed 20260803, torch 2.13.0, torch-dimensions 0.3.1. |
|
|
| One task — a cumulative sum along an axis — on one machine. That suits a |
| causal sequence model and is close to the worst case for a permutation- |
| invariant one, so this ranks models *on this problem* and implies nothing |
| about images or forecasting. There is no combined score on purpose: the |
| weighting between a point of loss and a step per second is the reader's, |
| not this file's. |
|
|
| | model | params | final loss | best | 90% at | tail spread | steps/s | Δloss per 1k params | |
| |---|---|---|---|---|---|---|---| |
| | `mamba_upstream_3d` | 50,752 | 0.0042 | 0.0035 | step 53 | 0.112 | 25.6 | 0.054 | |
| | `lstm_3d` | 51,136 | 0.0045 | 0.0011 | step 39 | 0.655 | 59.2 | 0.077 | |
| | `mamba3_2d` | 141,472 | 0.6098 | 0.4131 | step 22 | 0.122 | 12.0 | 0.026 | |
| | `gru_2d_sparse` | 38,464 | 0.6109 | 0.4073 | step 47 | 0.127 | 30.4 | 0.084 | |
| | `mamba_upstream_2d` | 33,856 | 0.6111 | 0.4048 | step 78 | 0.126 | 35.2 | 0.087 | |
| | `lstm_2d_sparse` | 51,136 | 0.6116 | 0.4030 | step 47 | 0.131 | 70.5 | 0.059 | |
| | `mamba_portable_2d` | 33,856 | 0.6124 | 0.4054 | step 78 | 0.126 | 39.9 | 0.087 | |
| | `mamba2_2d` | 111,920 | 0.6143 | 0.4340 | step 91 | 0.119 | 5.4 | 0.019 | |
| | `s4_upstream_2d` | 17,216 | 0.6227 | 0.4246 | step 85 | 0.124 | 35.5 | 0.152 | |
| | `s4d_upstream_2d` | 15,168 | 0.6252 | 0.4232 | step 85 | 0.123 | 41.3 | 0.175 | |
| | `s4d_portable_2d` | 13,120 | 0.6378 | 0.4188 | step 91 | 0.125 | 56.7 | 0.206 | |
| | `tcn_2d_sparse` | 25,152 | 1.0288 | 0.7823 | step 45 | 0.110 | 81.3 | 0.098 | |
| | `transformer_cafa_2d` | 55,376 | 1.4103 | 1.0995 | step 25 | 0.082 | 10.1 | 0.206 | |
| | `transformer_scan_2d` | 34,112 | 1.6962 | 1.3074 | step 74 | 0.080 | 49.8 | 0.049 | |
| | `cnn_2d_sparse` | 12,736 | 2.2291 | 1.5628 | step 45 | 0.106 | 97.7 | 0.140 | |
| | `transformer_flatten_2d` | 34,112 | 2.7320 | 1.9946 | step 164 | 0.104 | 35.7 | 0.021 | |
|
|
| ⚠ marks a model whose loss never fell by 5% of where it started. |
|
|
| ## Best on each question |
|
|
| - **lowest final loss** — `mamba_upstream_3d`, `lstm_3d`, `mamba3_2d` |
| - **fastest to 90% of its own improvement** — `mamba3_2d`, `transformer_cafa_2d`, `lstm_3d` |
| - **steadiest at the end** — `transformer_scan_2d`, `transformer_cafa_2d`, `transformer_flatten_2d` |
| - **most steps per second** — `cnn_2d_sparse`, `tcn_2d_sparse`, `lstm_2d_sparse` |
| - **most improvement per 1k parameters** — `s4d_portable_2d`, `transformer_cafa_2d`, `s4d_upstream_2d` |
|
|
