davidhouseholter commited on
Commit
9aaa591
·
verified ·
1 Parent(s): bdd6bea

Cross-link MatterWaveAI/grt-20m-curriculum in README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -21,7 +21,7 @@ pipeline_tag: text-generation
21
 
22
  # GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization
23
 
24
- > **Note:** `GRT-20M` (`16.7M` total parameters, `10.3M` non-embedding backbone parameters across `12` rotational blocks) was pretrained across an 8-corpus English mix using first-generation `PairAdam` for our NeurIPS submission (archived pre-fine-tune at [`MatterWaveAI/grt-20m-base`](https://huggingface.co/MatterWaveAI/grt-20m-base)). Keeping the exact same `16.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`38.63%` (`PairAdam`) → `45.03%` (`HuygensSymplecticAdam`, `+6.40%`)** and cuts `GSM8K` held-out math perplexity from **`60.12` → `9.11` (`6.60×` reduction vs. Zero-Shot, `-27.2%` vs. `PairAdam`)**.
25
 
26
  A Geometric Rotation Transformer pretrained on a mix of eight English corpora,
27
  including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear
 
21
 
22
  # GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization
23
 
24
+ > **Note:** `GRT-20M` (`16.7M` total parameters, `10.3M` non-embedding backbone parameters across `12` rotational blocks) was pretrained across an 8-corpus English mix using first-generation `PairAdam` for our NeurIPS submission (archived pre-fine-tune at [`MatterWaveAI/grt-20m-base`](https://huggingface.co/MatterWaveAI/grt-20m-base)). Keeping the exact same `16.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`38.63%` (`PairAdam`) → `45.03%` (`HuygensSymplecticAdam`, `+6.40%`)** and cuts `GSM8K` held-out math perplexity from **`60.12` → `9.11` (`6.60×` reduction vs. Zero-Shot, `-27.2%` vs. `PairAdam`)**. For the multi-round curriculum checkpoint (**`47.36%`** 9-task MC average, **`97.60%`** `GSM8K` verification, **`7.65`** PPL), see [`MatterWaveAI/grt-20m-curriculum`](https://huggingface.co/MatterWaveAI/grt-20m-curriculum).
25
 
26
  A Geometric Rotation Transformer pretrained on a mix of eight English corpora,
27
  including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear