Model card: add the independent CUDA replication (random run order)
Browse files
README.md
CHANGED
|
@@ -86,19 +86,22 @@ Use that loss to match the reference training.
|
|
| 86 |
|
| 87 |
Fine-tuning on the 61-word task with the authors' `run_cls.py` protocol (200
|
| 88 |
epochs, AdamW, sigmoid cross-entropy, 6 seeds × 12 subjects, paper-time data
|
| 89 |
-
revision `056a87b7`, mean ± SE over subjects).
|
| 90 |
-
was run with both the original model and the braindecode port, starting from
|
| 91 |
-
the same weights:
|
| 92 |
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
The port matches the authors' code to within 0.4 points. Training the port on
|
| 100 |
plain logits (`cross_entropy(logits)`) instead of the sigmoid loss gives 55.6
|
| 101 |
-
for mae.
|
| 102 |
|
| 103 |
**Split caveat.** The authors' split concatenates a subject's runs in
|
| 104 |
`os.listdir` order and puts the first 80 % of each word's trials in the
|
|
|
|
| 86 |
|
| 87 |
Fine-tuning on the 61-word task with the authors' `run_cls.py` protocol (200
|
| 88 |
epochs, AdamW, sigmoid cross-entropy, 6 seeds × 12 subjects, paper-time data
|
| 89 |
+
revision `056a87b7`, mean ± SE over subjects). There are two independent runs:
|
|
|
|
|
|
|
| 90 |
|
| 91 |
+
- **A**: the authors' `run_cls.train()` loop run with both the original model
|
| 92 |
+
and the port, starting from the same weights (Gaudi2 / CPU);
|
| 93 |
+
- **B**: a standalone braindecode re-implementation of the same loop with the
|
| 94 |
+
port, on CUDA, with the run order drawn at random per seed (see below).
|
| 95 |
+
|
| 96 |
+
| | paper (Table 2) | A: authors' code | A: port | B: port |
|
| 97 |
+
|---|---|---|---|---|
|
| 98 |
+
| Du-IN (mae) | 62.70 ± 4.69 | 60.52 ± 4.58 | 60.16 ± 4.62 | 59.37 ± 4.69 |
|
| 99 |
+
| Du-IN (vqvae) | 58.24 ± 4.83 | 55.10 ± 4.51 | 55.00 ± 4.51 | 54.02 ± 4.57 |
|
| 100 |
+
| scratch | 56.29 ± 5.20 | 54.06 ± 5.12 | 53.73 ± 5.13 | 53.77 ± 5.10 |
|
| 101 |
|
| 102 |
The port matches the authors' code to within 0.4 points. Training the port on
|
| 103 |
plain logits (`cross_entropy(logits)`) instead of the sigmoid loss gives 55.6
|
| 104 |
+
for mae in A and 54.5 in B.
|
| 105 |
|
| 106 |
**Split caveat.** The authors' split concatenates a subject's runs in
|
| 107 |
`os.listdir` order and puts the first 80 % of each word's trials in the
|