davidhouseholter commited on
Commit
1d0a50d
·
verified ·
1 Parent(s): 1e7aad5

Update GRT-10M card header to NeurIPS Submission baseline

Browse files
Files changed (1) hide show
  1. README.md +2 -3
README.md CHANGED
@@ -5,7 +5,6 @@ tags:
5
  - geometric-rotation-transformer
6
  - causal-lm
7
  - rotational
8
- - neurips-reject
9
  datasets:
10
  - nvidia/Nemotron-Pretraining-Dataset-sample
11
  language:
@@ -14,9 +13,9 @@ library_name: transformers
14
  pipeline_tag: text-generation
15
  ---
16
 
17
- # GRT-10M: From NeurIPS Reject (`PairAdam`) to Symplectic/Huygens Optimization
18
 
19
- > **Note:** `GRT-10M` (`8.7M` total parameters, `2.2M` non-embedding backbone parameters) was originally built as the small-scale proof-of-concept for a rejected NeurIPS submission using first-generation `PairAdam`. Keeping the exact same rejected `8.7M` base checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`37.61%` (`PairAdam`) $\rightarrow$ `41.79%` (`HuygensSymplecticAdam`)** and cuts `GSM8K` math perplexity from **`145.9` $\rightarrow$ `30.1` (`4.85×` reduction)**.
20
 
21
  A Geometric Rotation Transformer pretrained on Nemotron. GRT replaces the dense
22
  linear projections of a standard transformer with cascades of learned Givens
 
5
  - geometric-rotation-transformer
6
  - causal-lm
7
  - rotational
 
8
  datasets:
9
  - nvidia/Nemotron-Pretraining-Dataset-sample
10
  language:
 
13
  pipeline_tag: text-generation
14
  ---
15
 
16
+ # GRT-10M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization
17
 
18
+ > **Note:** `GRT-10M` (`8.7M` total parameters, `2.2M` non-embedding backbone parameters) was originally built as the small-scale proof-of-concept for our NeurIPS submission using first-generation `PairAdam`. Keeping the exact same `8.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`37.61%` (`PairAdam`) $\rightarrow$ `41.71%` (`HuygensSymplecticAdam`)** and cuts `GSM8K` math perplexity from **`145.9` $\rightarrow$ `30.1` (`4.85×` reduction)**.
19
 
20
  A Geometric Rotation Transformer pretrained on Nemotron. GRT replaces the dense
21
  linear projections of a standard transformer with cascades of learned Givens