Hugging Face's logo

MatterWaveAI
/
grt-10m

Text Generation
Transformers
Safetensors
English
grt
geometric-rotation-transformer
causal-lm
rotational
Model card Files
xet
 Community
davidhouseholter commited on
Commit
4979aa6
·
verified ·
1 Parent(s): 13337d5

Update SLM-Bench Round-1 curriculum results (46.00% macro avg)

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -64,17 +64,17 @@ Evaluated on the 10-task `SLM-Bench` small-language-model suite (`training/slm_b
64
 
65
  | Task (`SLM-Bench`) | Split (`n`) | Random | `GRT-10M` (Zero-Shot) | `PairAdam` | `SymplecticAdam` | `HuygensSymplecticAdam` | Delta (`HuygensSymplectic` vs. Zero-Shot) |
66
  |---|---|---:|---:|---:|---:|---:|---:|
67
- | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | **`62.14%`** | `61.13%` | **`62.14%`** | **`62.14%`** | `+0.00%` (`+12.14%` vs. random) |
68
- | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `26.94%` | `26.60%` | `31.73%` | **`32.03%`** | **`+5.09%`** |
69
- | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `22.53%` | `24.91%` | `25.17%` | **`26.62%`** | **`+4.09%`** |
70
- | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `30.00%` | `22.60%` | `33.40%` | **`34.00%`** | **`+4.00%`** |
71
  | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | **`52.18%`** | `50.82%` | **`52.18%`** | **`52.18%`** | `+0.00%` (`+2.18%` vs. random) |
72
  | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.51%` | `50.75%` | **`51.85%`** | **`51.85%`** | **`+1.34%`** |
73
  | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `25.60%` | `29.10%` | **`30.80%`** | **`30.80%`** | **`+5.20%`** |
74
  | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `52.70%` | `49.70%` | `56.80%` | **`60.90%`** | **`+8.20%`** |
75
  | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `20.15%` | **`22.85%`** | **`22.85%`** | **`22.85%`** | **`+2.70%`** |
76
  | **GSM8K** (`openai/gsm8k`, 4-way MC / PPL) | `test` (`500`) | `25.00%` | `22.00%` (`145.9` PPL) | `21.80%` (`37.3` PPL) | `79.40%` (`43.5` PPL) | **`84.60%`** (**`30.1` PPL**) | **`+62.60%`** (`-115.8` PPL) |
77
- | **10-Task Macro Average** | `13,644` total | `34.50%` | `36.47%` | `36.03%` | `44.63%` | **`45.80%`** | **`+9.33%`** (`+11.30%` vs. random) |
78
 
79
  Reproduce with `grt-slm-bench`:
80
 
 
64
 
65
  | Task (`SLM-Bench`) | Split (`n`) | Random | `GRT-10M` (Zero-Shot) | `PairAdam` | `SymplecticAdam` | `HuygensSymplecticAdam` | Delta (`HuygensSymplectic` vs. Zero-Shot) |
66
  |---|---|---:|---:|---:|---:|---:|---:|
67
+ | **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `62.14%` | `61.13%` | `62.14%` | **`62.17%`** | **`+0.03%`** (`+12.17%` vs. random) |
68
+ | **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `26.94%` | `26.60%` | `31.73%` | **`32.28%`** | **`+5.34%`** |
69
+ | **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `22.53%` | `24.91%` | `25.17%` | **`27.13%`** | **`+4.60%`** |
70
+ | **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `30.00%` | `22.60%` | `33.40%` | **`35.20%`** | **`+5.20%`** |
71
  | **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | **`52.18%`** | `50.82%` | **`52.18%`** | **`52.18%`** | `+0.00%` (`+2.18%` vs. random) |
72
  | **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `50.51%` | `50.75%` | **`51.85%`** | **`51.85%`** | **`+1.34%`** |
73
  | **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `25.60%` | `29.10%` | **`30.80%`** | **`30.80%`** | **`+5.20%`** |
74
  | **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `52.70%` | `49.70%` | `56.80%` | **`60.90%`** | **`+8.20%`** |
75
  | **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `20.15%` | **`22.85%`** | **`22.85%`** | **`22.85%`** | **`+2.70%`** |
76
  | **GSM8K** (`openai/gsm8k`, 4-way MC / PPL) | `test` (`500`) | `25.00%` | `22.00%` (`145.9` PPL) | `21.80%` (`37.3` PPL) | `79.40%` (`43.5` PPL) | **`84.60%`** (**`30.1` PPL**) | **`+62.60%`** (`-115.8` PPL) |
77
+ | **10-Task Macro Average** | `13,644` total | `34.50%` | `36.47%` | `36.03%` | `44.63%` | **`46.00%`** | **`+9.53%`** (`+11.50%` vs. random) |
78
 
79
  Reproduce with `grt-slm-bench`:
80