davidhouseholter commited on
Commit
8f86b55
·
verified ·
1 Parent(s): e42bbd1

Update MatterWaveAI/grt-20m-curriculum final 24h metrics in README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -6
README.md CHANGED
@@ -21,7 +21,7 @@ pipeline_tag: text-generation
21
 
22
  # GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization
23
 
24
- > **Note:** `GRT-20M` (`16.7M` total parameters, `10.3M` non-embedding backbone parameters across `12` rotational blocks) was pretrained across an 8-corpus English mix using first-generation `PairAdam` for our NeurIPS submission (archived pre-fine-tune at [`MatterWaveAI/grt-20m-base`](https://huggingface.co/MatterWaveAI/grt-20m-base)). Keeping the exact same `16.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`38.63%` (`PairAdam`) → `45.03%` (`HuygensSymplecticAdam`, `+6.40%`)** and cuts `GSM8K` held-out math perplexity from **`60.12` → `9.11` (`6.60×` reduction vs. Zero-Shot, `-27.2%` vs. `PairAdam`)**. For the multi-round curriculum checkpoint (**`47.36%`** 9-task MC average, **`97.60%`** `GSM8K` verification, **`7.65`** PPL), see [`MatterWaveAI/grt-20m-curriculum`](https://huggingface.co/MatterWaveAI/grt-20m-curriculum).
25
 
26
  A Geometric Rotation Transformer pretrained on a mix of eight English corpora,
27
  including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear
@@ -59,11 +59,12 @@ plus a log-magnitude per token).
59
 
60
  | | |
61
  |---|---|
62
- | **Data** | ClimbMix, Wikipedia, RefinedWeb, FineWeb-Edu, Cosmopedia, Nemotron, FineMath, StackExchange |
63
- | **Steps** | 158,000 |
64
- | **Optimizer** | PairAdam |
65
- | **Peak LR** | 1e-3, cosine decay to 10% |
66
- | **Training loss** | 2.114 |
 
67
 
68
  ## Sample output (greedy, temperature 0)
69
 
 
21
 
22
  # GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization
23
 
24
+ > **Note:** `GRT-20M` (`16.7M` total parameters, `10.3M` non-embedding backbone parameters across `12` rotational blocks) was pretrained across an 8-corpus English mix using first-generation `PairAdam` for our NeurIPS submission (archived pre-fine-tune at [`MatterWaveAI/grt-20m-base`](https://huggingface.co/MatterWaveAI/grt-20m-base)). Keeping the exact same `16.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`38.63%` (`PairAdam`) → `45.03%` (`HuygensSymplecticAdam`, `+6.40%`)** and cuts `GSM8K` held-out math perplexity from **`60.12` → `9.11` (`6.60×` reduction vs. Zero-Shot, `-27.2%` vs. `PairAdam`)**. For the multi-round curriculum checkpoint (**`48.17%`** 9-task MC average, **`98.20%`** `GSM8K` verification, **`7.65`** PPL), see [`MatterWaveAI/grt-20m-curriculum`](https://huggingface.co/MatterWaveAI/grt-20m-curriculum).
25
 
26
  A Geometric Rotation Transformer pretrained on a mix of eight English corpora,
27
  including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear
 
59
 
60
  | | |
61
  |---|---|
62
+ | **Base Pretraining Data** | ClimbMix, Wikipedia, RefinedWeb, FineWeb-Edu, Cosmopedia, Nemotron, FineMath, StackExchange |
63
+ | **Base Pretraining Steps** | 158,000 |
64
+ | **Base Pretraining Optimizer** | PairAdam (`MatterWaveAI/grt-20m-base`) |
65
+ | **Base Peak LR** | 1e-3, cosine decay to 10% |
66
+ | **Base Training Loss** | 2.114 |
67
+ | **Task Adaptation Optimizer** | `HuygensSymplecticAdam` (`SymplecticAdam` two-stage curriculum) |
68
 
69
  ## Sample output (greedy, temperature 0)
70