Upcycle GRT-20M with 10-task SLM-Bench evaluation (Zero-Shot, PairAdam, SymplecticAdam, HuygensSymplecticAdam)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+grt-20m-f32.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,223 @@
+---
+license: apache-2.0
+tags:
+  - grt
+  - geometric-rotation-transformer
+  - causal-lm
+  - rotational
+datasets:
+  - OptimalScale/ClimbMix
+  - wikimedia/wikipedia
+  - tiiuae/falcon-refinedweb
+  - HuggingFaceFW/fineweb-edu
+  - HuggingFaceTB/cosmopedia
+  - nvidia/Nemotron-Pretraining-Dataset-sample
+  - HuggingFaceTB/finemath
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization
+
+> **Note:** `GRT-20M` (`16.7M` total parameters, `10.3M` non-embedding backbone parameters across `12` rotational blocks) was pretrained across an 8-corpus English mix using first-generation `PairAdam` for our NeurIPS submission (archived pre-fine-tune at [`MatterWaveAI/grt-20m-base`](https://huggingface.co/MatterWaveAI/grt-20m-base)). Keeping the exact same `16.7M` NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (`SymplecticAdam` and `HuygensSymplecticAdam`) lifts 9-task `SLM-Bench` multiple-choice accuracy from **`38.63%` (`PairAdam`) $\rightarrow$ `45.03%` (`HuygensSymplecticAdam`, `+6.40%`)** and cuts `GSM8K` held-out math perplexity from **`60.12` $\rightarrow$ `9.11` (`6.60×` reduction vs. Zero-Shot, `-27.2%` vs. `PairAdam`)**.
+
+A Geometric Rotation Transformer pretrained on a mix of eight English corpora,
+including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear
+projections of a standard transformer with cascades of learned Givens rotations
+on SO(D), so the residual path contains no `nn.Linear` at all.
+
+**This is a research preview, not a product model.** It exists to show that the
+architecture trains stably and evaluates end-to-end through the same OpenAI-HTTP
+serving seam as every other model, and to give CPU-only CI a cheap, always-
+available upstream. At this scale the output is grammatical and locally
+coherent but not factual — see the real samples below before expecting anything
+else. It is not instruction-tuned and has no chat template.
+
+## Model Details
+
+| | |
+|---|---|
+| **Total parameters** | 16,730,869 (16.7M) |
+| **Non-embedding parameters** | 10,280,869 (10.3M) |
+| **d_model** | 256 |
+| **n_blocks** | 12 |
+| **n_heads** | 4 |
+| **n_qkv_rotations** | 11 |
+| **ffn_n_rotations** | 21 |
+| **Vocabulary** | 50,000 (BPE) |
+| **Max sequence length** | 1,024 |
+| **Precision** | float32 |
+
+Parameter counts are measured from the published `model.safetensors`. "20m" is
+the size-class label, not a parameter count. The embedding dominates: 6.45M of
+16.7M parameters are the token embedding (50,000 tokens x 128 phase angles,
+plus a log-magnitude per token).
+
+## Training
+
+| | |
+|---|---|
+| **Data** | ClimbMix, Wikipedia, RefinedWeb, FineWeb-Edu, Cosmopedia, Nemotron, FineMath, StackExchange |
+| **Steps** | 158,000 |
+| **Optimizer** | PairAdam |
+| **Peak LR** | 1e-3, cosine decay to 10% |
+| **Training loss** | 2.114 |
+
+## Sample output (greedy, temperature 0)
+
+The three reference completions, generated by the published weights through
+`transformers` f32, and reproduced token-for-token identically by the vLLM and
+llama.cpp serving paths:
+
+| Prompt | Greedy continuation (20 tokens) |
+|---|---|
+| `The capital of France is` | ` the capital of the capital of the capital of the capital of the capital of the capital of the capital` |
+| `Once upon a time` | `, there was a great idea to help us understand the complexities of the concept of the concept of the` |
+| `In the beginning` | `, the first term is the same, but the second term is the same.\n\nThe problem` |
+
+This is what the model does: locally plausible, globally wrong. That is the
+expected behaviour of a 16.7M base model at loss 2.114, not a serving bug.
+
+## Benchmarks: NeurIPS Submission Baseline (`PairAdam`) vs. New Optimizer (`HuygensSymplecticAdam`)
+
+Evaluated on the `SLM-Bench` small-language-model suite (`training/slm_bench.py` / `grt-slm-bench`) starting from the exact same `16.7M`-parameter (`10.3M` non-embedding) NeurIPS submission checkpoint:
+
+### 1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)
+
+| Task (`SLM-Bench`) | Split (`n`) | Random | `GRT-20M` Zero-Shot | NeurIPS Baseline (`PairAdam`) | `SymplecticAdam` | New Optimizer (`HuygensSymplecticAdam`) | Gain vs. `PairAdam` |
+|---|---|---:|---:|---:|---:|---:|---:|
+| **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | `50.00%` | `50.50%` | `55.70%` | `74.00%` | **`78.20%`** | **`+22.50%`** (`+28.20%` vs. random) |
+| **BoolQ** (`google/boolq`) | `val` (`3,270`) | `50.00%` | `62.17%` | `53.43%` | `62.17%` | **`62.35%`** | **`+8.92%`** (`+12.35%` vs. random) |
+| **PIQA** (`baber/piqa`) | `val` (`1,838`) | `50.00%` | `53.97%` | `53.37%` | **`53.97%`** | **`53.97%`** | **`+0.60%`** (`+3.97%` vs. random) |
+| **WinoGrande** (`allenai/winogrande`) | `val` (`1,267`) | `50.00%` | `51.93%` | `52.41%` | **`52.41%`** | **`52.41%`** | **`+0.48%`** vs. zero-shot |
+| **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | `25.00%` | `34.20%` | `25.60%` | `36.00%` | **`37.20%`** | **`+11.60%`** (`+12.20%` vs. random) |
+| **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | `25.00%` | `26.43%` | `26.85%` | `34.30%` | **`36.70%`** | **`+9.85%`** (`+11.70%` vs. random) |
+| **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | `25.00%` | `28.70%` | `31.80%` | **`32.70%`** | **`32.70%`** | **`+0.90%`** (`+7.70%` vs. random) |
+| **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `25.00%` | `25.34%` | `25.43%` | `26.79%` | **`27.13%`** | **`+1.70%`** (`+2.13%` vs. random) |
+| **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | `20.00%` | `19.16%` | `23.10%` | `23.91%` | **`24.57%`** | **`+1.47%`** (`+5.41%` vs. zero-shot) |
+| **9-Task Multiple-Choice Macro Average** | `13,144` total | `35.56%` | `39.16%` | `38.63%` | `44.03%` | **`45.03%`** | **`+6.40%`** (`+9.47%` vs. random) |
+
+### 2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`)
+
+| Task (`SLM-Bench`) | Split (`n`) | `GRT-20M` Zero-Shot | NeurIPS Baseline (`PairAdam`) | `SymplecticAdam` | New Optimizer (`HuygensSymplecticAdam`) | Improvement |
+|---|---|---:|---:|---:|---:|---:|
+| **GSM8K Held-Out Perplexity (`PPL` $\downarrow$)** | `test` (`500`) | `60.12` | `12.52` | `10.70` | **`9.11`** | **`6.60×` lower vs. Zero-Shot (`-27.2%` vs. `PairAdam`)** |
+| **GSM8K 4-Way Numerical Verification (`Acc` $\uparrow$)** | `test` (`500`) | `21.00%` | `28.80%` | `91.40%` | **`93.20%`** | **`+64.40%` vs. `PairAdam`** |
+
+Reproduce with `grt-slm-bench`:
+
+```bash
+python -m training.slm_bench --checkpoint ./ --optimizer huygens_symplectic_adam \
+    --tasks boolq,arc_easy,arc_challenge,openbookqa,piqa,winogrande,hellaswag,qnli,commonsenseqa,gsm8k \
+    --device cuda
+```
+
+Zero-shot, via lm-evaluation-harness:
+
+```bash
+lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-20m \
+    --tasks boolq,arc_easy --batch_size 64
+```
+
+## Usage
+
+```python
+import torch
+import grt  # registers model_type "grt" with the transformers Auto classes
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-20m", dtype=torch.float32)
+tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-20m")
+
+input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids
+output = model.generate(input_ids, max_new_tokens=50, do_sample=False)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional
+and load automatically when installed.
+
+> **`trust_remote_code=True` is not supported for this repository.** The model
+> code is distributed as a pip package rather than as Hub remote code, because
+> the implementation spans a package with absolute imports that a flat Hub
+> snapshot cannot resolve. Install `geometric-rotation-transformer` first;
+> `import grt` then registers `model_type: "grt"` so the Auto classes resolve
+> these weights.
+
+### GGUF
+
+`grt-20m-f32.gguf` is the same weights in GGUF form, loadable by the MatterWave
+llama.cpp fork. Verify any pull against `SHA256SUMS` in this repo:
+
+```bash
+sha256sum --ignore-missing -c SHA256SUMS
+```
+
+## Architecture
+
+GRT replaces the standard Q/K/V projections and FFN weight matrices with
+cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is
+modulated by data-dependent Clifford features:
+
+```
+feat  = [S0, V1, V2, B12]           # scalar energy, direction, cross-pair bivector
+theta = angles + tanh(modulation)   # base angle + content-dependent nudge
+r0    = x0*cos(theta) - x1*sin(theta)
+r1    = x0*sin(theta) + x1*cos(theta)
+```
+
+Components:
+
+- **RotationalEmbedding** — tokens stored as phase angles on the unit hypersphere
+- **RotationalAttention** — Q/K/V via Givens cascades
+- **RotationalFFN** — feed-forward via meta-coupled rotation stages
+- **PearlNorm** — pair-radius normalization preserving per-pair phase
+- **RadialGELU** — activation gating pair magnitudes, preserving direction
+- **PairAdam** — AdamW with factored second moments: 1D params (angles, scales)
+  use standard diagonal moments, 2D params (coupling matrices) use a row factor
+  [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n)
+  instead of O(mn). Optionally clamps per-step angular updates.
+
+Because every projection is a rotation, the transform is norm-preserving by
+construction rather than by normalization.
+
+## Intended Use & Limitations
+
+Intended for research into structured/rotational architectures and as a
+deterministic CI upstream. Not intended for production use, deployment, or any
+task requiring factual output.
+
+Capabilities this model does not have — the platform metadata reflects this
+(`supports_tools: false`, `supports_response_schema: false`,
+`supports_vision: false`):
+
+- **No instruction tuning.** This is a base model trained on next-token
+  prediction only.
+- **No chat template.** There is no `chat_template` in this repo; an
+  OpenAI-compatible `/v1/chat/completions` endpoint serves it by concatenating
+  the messages, which is fine for a demo and wrong for anything else.
+- **No tool calling, no structured/schema-constrained output, no vision.**
+- English only. The mix includes web crawl (RefinedWeb, ClimbMix), encyclopedic
+  text, synthetic textbooks (Cosmopedia), math (FineMath) and Q&A
+  (StackExchange), so it inherits the coverage gaps and biases of all of them.
+- Generated text is frequently incoherent past a clause or two. Do not expect
+  instruction-following, factual reliability, or coherent long-form output.
+- No safety tuning, RLHF, or content filtering of any kind.
+- Context is capped at 1,024 tokens.
+
+## Citation
+
+```bibtex
+@article{grt2026,
+  title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling},
+  author={MatterWave AI},
+  year={2026},
+  url={https://github.com/matterwaveai/grt}
+}
+```
+
+## License
+
+Apache 2.0

SHA256SUMS

@@ -0,0 +1,9 @@
+1e3367c85467c08f0e1c64e0c46cb513f4c0cca46ad99061ee6b8202db82e6ad  config.json
+638762238fc4bf981e6ccdd02293fe30f2f11b07e240d8dcdca2263b1b0f611d  eval_results.json
+f80e3100d8af7f3a6b38d298de8246f6edc682ad742965e8bcab05da2ea30206  generation_config.json
+e6e74fed83a4ec3b2f5e640697295ab4a80615057b62d3c131c74149826db8bc  grt-20m-f32.gguf
+93ac9e8d03a592cf0b98fde1f473cbe03a05e01c9c747bc90d8e7e2e5a427aa3  model.safetensors
+dd8c5db1b3e3272dd1542a8811d462dc9e181f4b38a20b4f6cb69eb7030f9ec8  README.md
+c116ee9b68688619ebd08889e66408ff255499b0490cd7e93eccb5a09dd548df  slm_bench_results.json
+fc0fab0e6fad9006677fe4a1161e7538acf1c644464d04e810f64ba5901c473e  tokenizer.json
+c86a1ad063b81c4f76897970fad6a92df5007fd6644e4805ef2c99265c205cb4  tokenizer_config.json

config.json

@@ -0,0 +1,30 @@
+{
+  "angle_init_scale": 0.1,
+  "architectures": [
+    "GRTForCausalLM"
+  ],
+  "bos_token_id": 2,
+  "coupling_init_scale": 0.01,
+  "d_ff": null,
+  "d_model": 256,
+  "dropout": 0.0,
+  "eos_token_id": 3,
+  "ffn_mult": 4,
+  "ffn_n_rotations": 21,
+  "learnable_rope": true,
+  "log_magnitude_init": 0.0,
+  "max_seq_len": 1024,
+  "model_type": "grt",
+  "n_blocks": 12,
+  "n_heads": 4,
+  "n_mixer_rotations": 1,
+  "n_qkv_rotations": 11,
+  "num_hidden_layers": 12,
+  "output_scale_init": 30.0,
+  "pad_token_id": 0,
+  "phase_init_scale": 0.1,
+  "pos_freq_base": 10000.0,
+  "residual_gate_mode": "bounded_depth",
+  "transformers_version": "4.47.1",
+  "vocab_size": 50000
+}

eval_results.json

@@ -0,0 +1,137 @@
+{
+  "model_id": "MatterWaveAI/grt-20m",
+  "architecture": "GRTForCausalLM",
+  "stored_parameters": 16730869,
+  "non_embedding_parameters": 10280869,
+  "dense_equiv_parameters": 22237440,
+  "nn_linear_layers": 0,
+  "pretraining": {
+    "corpus": [
+      "OptimalScale/ClimbMix",
+      "wikimedia/wikipedia",
+      "tiiuae/falcon-refinedweb",
+      "HuggingFaceFW/fineweb-edu",
+      "HuggingFaceTB/cosmopedia",
+      "nvidia/Nemotron-Pretraining-Dataset-sample",
+      "HuggingFaceTB/finemath"
+    ],
+    "steps": 158000,
+    "optimizer": "PairAdam",
+    "train_loss": 2.114
+  },
+  "slm_bench": {
+    "benchmark_suite": "SLM-Bench (10-Task Core Suite)",
+    "evaluation_harness": "grt-slm-bench (training/slm_bench.py)",
+    "hardware": "AMD Ryzen AI Max+ 395 (Radeon 8060S / ROCm gfx1151)",
+    "mc_9task_macro_average_accuracy": {
+      "random_baseline": 0.3556,
+      "zero_shot": 0.3916,
+      "pair_adam": 0.3863,
+      "symplectic_adam": 0.4403,
+      "huygens_symplectic_adam": 0.4503
+    },
+    "macro_average_accuracy": {
+      "random_baseline": 0.345,
+      "zero_shot": 0.3734,
+      "pair_adam": 0.3765,
+      "symplectic_adam": 0.4877,
+      "huygens_symplectic_adam": 0.4984
+    },
+    "tasks": {
+      "boolq": {
+        "split": "validation",
+        "n_eval": 3270,
+        "random_baseline": 0.5,
+        "zero_shot_acc": 0.6217,
+        "pair_adam_acc": 0.5343,
+        "symplectic_adam_acc": 0.6217,
+        "huygens_symplectic_adam_acc": 0.6235
+      },
+      "arc_easy": {
+        "split": "test",
+        "n_eval": 2376,
+        "random_baseline": 0.25,
+        "zero_shot_acc": 0.2643,
+        "pair_adam_acc": 0.2685,
+        "symplectic_adam_acc": 0.343,
+        "huygens_symplectic_adam_acc": 0.367
+      },
+      "arc_challenge": {
+        "split": "test",
+        "n_eval": 1172,
+        "random_baseline": 0.25,
+        "zero_shot_acc": 0.2534,
+        "pair_adam_acc": 0.2543,
+        "symplectic_adam_acc": 0.2679,
+        "huygens_symplectic_adam_acc": 0.2713
+      },
+      "openbookqa": {
+        "split": "test",
+        "n_eval": 500,
+        "random_baseline": 0.25,
+        "zero_shot_acc": 0.342,
+        "pair_adam_acc": 0.256,
+        "symplectic_adam_acc": 0.36,
+        "huygens_symplectic_adam_acc": 0.372
+      },
+      "piqa": {
+        "split": "validation",
+        "n_eval": 1838,
+        "random_baseline": 0.5,
+        "zero_shot_acc": 0.5397,
+        "pair_adam_acc": 0.5337,
+        "symplectic_adam_acc": 0.5397,
+        "huygens_symplectic_adam_acc": 0.5397
+      },
+      "winogrande": {
+        "split": "validation",
+        "n_eval": 1267,
+        "random_baseline": 0.5,
+        "zero_shot_acc": 0.5193,
+        "pair_adam_acc": 0.5241,
+        "symplectic_adam_acc": 0.5241,
+        "huygens_symplectic_adam_acc": 0.5241
+      },
+      "hellaswag": {
+        "split": "validation",
+        "n_eval": 1000,
+        "random_baseline": 0.25,
+        "zero_shot_acc": 0.287,
+        "pair_adam_acc": 0.318,
+        "symplectic_adam_acc": 0.327,
+        "huygens_symplectic_adam_acc": 0.327
+      },
+      "qnli": {
+        "split": "validation",
+        "n_eval": 1000,
+        "random_baseline": 0.5,
+        "zero_shot_acc": 0.505,
+        "pair_adam_acc": 0.557,
+        "symplectic_adam_acc": 0.74,
+        "huygens_symplectic_adam_acc": 0.782
+      },
+      "commonsenseqa": {
+        "split": "validation",
+        "n_eval": 1221,
+        "random_baseline": 0.2,
+        "zero_shot_acc": 0.1916,
+        "pair_adam_acc": 0.231,
+        "symplectic_adam_acc": 0.2391,
+        "huygens_symplectic_adam_acc": 0.2457
+      },
+      "gsm8k": {
+        "split": "test",
+        "n_eval": 500,
+        "random_baseline": 0.25,
+        "zero_shot_acc": 0.21,
+        "zero_shot_ppl": 60.12,
+        "pair_adam_acc": 0.288,
+        "pair_adam_ppl": 12.52,
+        "symplectic_adam_acc": 0.914,
+        "symplectic_adam_ppl": 10.7,
+        "huygens_symplectic_adam_acc": 0.932,
+        "huygens_symplectic_adam_ppl": 9.11
+      }
+    }
+  }
+}

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "pad_token_id": 0
+}

grt-20m-f32.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6e74fed83a4ec3b2f5e640697295ab4a80615057b62d3c131c74149826db8bc
+size 66576224

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93ac9e8d03a592cf0b98fde1f473cbe03a05e01c9c747bc90d8e7e2e5a427aa3
+size 69119972

slm_bench_results.json

@@ -0,0 +1,383 @@
+{
+  "model_id": "MatterWaveAI/grt-20m",
+  "architecture": "GRTForCausalLM",
+  "stored_parameters": 16190197,
+  "dense_equiv_parameters": 22237440,
+  "benchmark_suite": "SLM-Bench (10-Task Core Suite)",
+  "evaluation_harness": "grt-slm-bench (training/slm_bench.py)",
+  "hardware": "AMD Ryzen AI Max+ 395 (Radeon 8060S / ROCm gfx1151)",
+  "total_wall_clock_sec": 4270.55,
+  "macro_average_accuracy": {
+    "random_baseline": 0.345,
+    "zero_shot": 0.3734,
+    "pair_adam": 0.3765,
+    "symplectic_adam": 0.4877,
+    "huygens_symplectic_adam": 0.4984
+  },
+  "tasks": {
+    "boolq": {
+      "task": "boolq",
+      "dataset": "google/boolq",
+      "eval_split": "validation",
+      "n_eval": 3270,
+      "random_baseline": 0.5,
+      "zero_shot": {
+        "accuracy": 0.6217125382262997,
+        "n_eval": 3270,
+        "eval_sec": 72.68
+      },
+      "pair_adam": {
+        "accuracy": 0.5342507645259938,
+        "n_eval": 3270,
+        "eval_sec": 72.91,
+        "finetune_sec": 54.45,
+        "finetune_steps": 80,
+        "best_train_loss": 2.6083
+      },
+      "symplectic_adam": {
+        "accuracy": 0.6217125382262997,
+        "n_eval": 3270,
+        "eval_sec": 72.68,
+        "finetune_sec": 326.46,
+        "finetune_steps": 120,
+        "best_train_loss": 0.3416
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.6235474006116208,
+        "n_eval": 3270,
+        "eval_sec": 72.72,
+        "finetune_sec": 417.9,
+        "finetune_steps": 140,
+        "best_train_loss": 0.237
+      }
+    },
+    "arc_easy": {
+      "task": "arc_easy",
+      "dataset": "allenai/ai2_arc:ARC-Easy",
+      "eval_split": "test",
+      "n_eval": 2376,
+      "random_baseline": 0.25,
+      "zero_shot": {
+        "accuracy": 0.26430976430976433,
+        "n_eval": 2376,
+        "eval_sec": 33.52
+      },
+      "pair_adam": {
+        "accuracy": 0.26851851851851855,
+        "n_eval": 2376,
+        "eval_sec": 33.69,
+        "finetune_sec": 52.26,
+        "finetune_steps": 80,
+        "best_train_loss": 1.7257
+      },
+      "symplectic_adam": {
+        "accuracy": 0.343013468013468,
+        "n_eval": 2376,
+        "eval_sec": 33.76,
+        "finetune_sec": 200.17,
+        "finetune_steps": 120,
+        "best_train_loss": 0.793
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.367003367003367,
+        "n_eval": 2376,
+        "eval_sec": 33.64,
+        "finetune_sec": 249.23,
+        "finetune_steps": 140,
+        "best_train_loss": 0.7056
+      }
+    },
+    "arc_challenge": {
+      "task": "arc_challenge",
+      "dataset": "allenai/ai2_arc:ARC-Challenge",
+      "eval_split": "test",
+      "n_eval": 1172,
+      "random_baseline": 0.25,
+      "zero_shot": {
+        "accuracy": 0.25341296928327645,
+        "n_eval": 1172,
+        "eval_sec": 19.74
+      },
+      "pair_adam": {
+        "accuracy": 0.25426621160409557,
+        "n_eval": 1172,
+        "eval_sec": 19.93,
+        "finetune_sec": 52.62,
+        "finetune_steps": 80,
+        "best_train_loss": 1.5342
+      },
+      "symplectic_adam": {
+        "accuracy": 0.26791808873720135,
+        "n_eval": 1172,
+        "eval_sec": 19.92,
+        "finetune_sec": 170.58,
+        "finetune_steps": 120,
+        "best_train_loss": 0.6139
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.2713310580204778,
+        "n_eval": 1172,
+        "eval_sec": 19.87,
+        "finetune_sec": 209.72,
+        "finetune_steps": 140,
+        "best_train_loss": 0.4093
+      }
+    },
+    "openbookqa": {
+      "task": "openbookqa",
+      "dataset": "allenai/openbookqa:main",
+      "eval_split": "test",
+      "n_eval": 500,
+      "random_baseline": 0.25,
+      "zero_shot": {
+        "accuracy": 0.342,
+        "n_eval": 500,
+        "eval_sec": 4.55
+      },
+      "pair_adam": {
+        "accuracy": 0.256,
+        "n_eval": 500,
+        "eval_sec": 4.69,
+        "finetune_sec": 51.98,
+        "finetune_steps": 80,
+        "best_train_loss": 2.4451
+      },
+      "symplectic_adam": {
+        "accuracy": 0.36,
+        "n_eval": 500,
+        "eval_sec": 4.7,
+        "finetune_sec": 81.83,
+        "finetune_steps": 120,
+        "best_train_loss": 0.8312
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.372,
+        "n_eval": 500,
+        "eval_sec": 4.7,
+        "finetune_sec": 98.47,
+        "finetune_steps": 140,
+        "best_train_loss": 0.6025
+      }
+    },
+    "piqa": {
+      "task": "piqa",
+      "dataset": "baber/piqa",
+      "eval_split": "validation",
+      "n_eval": 1838,
+      "random_baseline": 0.5,
+      "zero_shot": {
+        "accuracy": 0.5397170837867247,
+        "n_eval": 1838,
+        "eval_sec": 13.95
+      },
+      "pair_adam": {
+        "accuracy": 0.5337323177366703,
+        "n_eval": 1838,
+        "eval_sec": 14.0,
+        "finetune_sec": 49.49,
+        "finetune_steps": 80,
+        "best_train_loss": 4.9004
+      },
+      "symplectic_adam": {
+        "accuracy": 0.5397170837867247,
+        "n_eval": 1838,
+        "eval_sec": 13.95,
+        "finetune_sec": 86.21,
+        "finetune_steps": 120,
+        "best_train_loss": 0.8876
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.5397170837867247,
+        "n_eval": 1838,
+        "eval_sec": 13.95,
+        "finetune_sec": 108.54,
+        "finetune_steps": 140,
+        "best_train_loss": 0.8785
+      }
+    },
+    "winogrande": {
+      "task": "winogrande",
+      "dataset": "allenai/winogrande:winogrande_xl",
+      "eval_split": "validation",
+      "n_eval": 1267,
+      "random_baseline": 0.5,
+      "zero_shot": {
+        "accuracy": 0.5193370165745856,
+        "n_eval": 1267,
+        "eval_sec": 3.45
+      },
+      "pair_adam": {
+        "accuracy": 0.5240726124704025,
+        "n_eval": 1267,
+        "eval_sec": 3.09,
+        "finetune_sec": 53.91,
+        "finetune_steps": 80,
+        "best_train_loss": 4.1441
+      },
+      "symplectic_adam": {
+        "accuracy": 0.5240726124704025,
+        "n_eval": 1267,
+        "eval_sec": 3.09,
+        "finetune_sec": 33.62,
+        "finetune_steps": 120,
+        "best_train_loss": 0.7208
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.5240726124704025,
+        "n_eval": 1267,
+        "eval_sec": 3.09,
+        "finetune_sec": 42.17,
+        "finetune_steps": 140,
+        "best_train_loss": 0.8362
+      }
+    },
+    "hellaswag": {
+      "task": "hellaswag",
+      "dataset": "Rowan/hellaswag",
+      "eval_split": "validation",
+      "n_eval": 1000,
+      "random_baseline": 0.25,
+      "zero_shot": {
+        "accuracy": 0.287,
+        "n_eval": 1000,
+        "eval_sec": 10.17
+      },
+      "pair_adam": {
+        "accuracy": 0.318,
+        "n_eval": 1000,
+        "eval_sec": 10.4,
+        "finetune_sec": 53.98,
+        "finetune_steps": 80,
+        "best_train_loss": 4.4042
+      },
+      "symplectic_adam": {
+        "accuracy": 0.327,
+        "n_eval": 1000,
+        "eval_sec": 10.41,
+        "finetune_sec": 110.37,
+        "finetune_steps": 120,
+        "best_train_loss": 0.7104
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.327,
+        "n_eval": 1000,
+        "eval_sec": 10.41,
+        "finetune_sec": 132.41,
+        "finetune_steps": 140,
+        "best_train_loss": 1.1398
+      }
+    },
+    "qnli": {
+      "task": "qnli",
+      "dataset": "nyu-mll/glue:qnli",
+      "eval_split": "validation",
+      "n_eval": 1000,
+      "random_baseline": 0.5,
+      "zero_shot": {
+        "accuracy": 0.505,
+        "n_eval": 1000,
+        "eval_sec": 7.59
+      },
+      "pair_adam": {
+        "accuracy": 0.557,
+        "n_eval": 1000,
+        "eval_sec": 7.8,
+        "finetune_sec": 54.27,
+        "finetune_steps": 80,
+        "best_train_loss": 2.7277
+      },
+      "symplectic_adam": {
+        "accuracy": 0.74,
+        "n_eval": 1000,
+        "eval_sec": 7.81,
+        "finetune_sec": 73.1,
+        "finetune_steps": 120,
+        "best_train_loss": 0.2444
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.782,
+        "n_eval": 1000,
+        "eval_sec": 7.84,
+        "finetune_sec": 91.88,
+        "finetune_steps": 140,
+        "best_train_loss": 0.148
+      }
+    },
+    "commonsenseqa": {
+      "task": "commonsenseqa",
+      "dataset": "tau/commonsense_qa",
+      "eval_split": "validation",
+      "n_eval": 1221,
+      "random_baseline": 0.2,
+      "zero_shot": {
+        "accuracy": 0.19164619164619165,
+        "n_eval": 1221,
+        "eval_sec": 11.92
+      },
+      "pair_adam": {
+        "accuracy": 0.23095823095823095,
+        "n_eval": 1221,
+        "eval_sec": 12.13,
+        "finetune_sec": 54.01,
+        "finetune_steps": 80,
+        "best_train_loss": 3.9139
+      },
+      "symplectic_adam": {
+        "accuracy": 0.23914823914823916,
+        "n_eval": 1221,
+        "eval_sec": 12.16,
+        "finetune_sec": 112.69,
+        "finetune_steps": 120,
+        "best_train_loss": 1.2686
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.2457002457002457,
+        "n_eval": 1221,
+        "eval_sec": 12.14,
+        "finetune_sec": 138.32,
+        "finetune_steps": 140,
+        "best_train_loss": 1.1876
+      }
+    },
+    "gsm8k": {
+      "task": "gsm8k",
+      "dataset": "openai/gsm8k:main",
+      "eval_split": "test",
+      "n_eval": 500,
+      "random_baseline": 0.25,
+      "zero_shot": {
+        "accuracy": 0.21,
+        "perplexity": 60.1165885925293,
+        "n_eval": 500,
+        "eval_sec": 24.2
+      },
+      "pair_adam": {
+        "accuracy": 0.288,
+        "perplexity": 12.520118713378906,
+        "n_eval": 500,
+        "eval_sec": 24.73,
+        "finetune_sec": 57.21,
+        "finetune_steps": 80,
+        "best_train_loss": 2.2787
+      },
+      "symplectic_adam": {
+        "accuracy": 0.914,
+        "perplexity": 10.70208740234375,
+        "n_eval": 500,
+        "eval_sec": 24.71,
+        "finetune_sec": 266.46,
+        "finetune_steps": 120,
+        "best_train_loss": 0.442
+      },
+      "huygens_symplectic_adam": {
+        "accuracy": 0.932,
+        "perplexity": 9.112833023071289,
+        "n_eval": 500,
+        "eval_sec": 24.71,
+        "finetune_sec": 325.33,
+        "finetune_steps": 140,
+        "best_train_loss": 0.3663
+      }
+    }
+  }
+}

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "unk_token": "<unk>",
+  "pad_token": "<pad>",
+  "model_max_length": 1024
+}