Instructions to use MatterWaveAI/grt-20m-curriculum with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MatterWaveAI/grt-20m-curriculum with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MatterWaveAI/grt-20m-curriculum")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-20m-curriculum", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MatterWaveAI/grt-20m-curriculum with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MatterWaveAI/grt-20m-curriculum"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MatterWaveAI/grt-20m-curriculum",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MatterWaveAI/grt-20m-curriculum

SGLang

How to use MatterWaveAI/grt-20m-curriculum with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MatterWaveAI/grt-20m-curriculum" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MatterWaveAI/grt-20m-curriculum",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MatterWaveAI/grt-20m-curriculum" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MatterWaveAI/grt-20m-curriculum",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MatterWaveAI/grt-20m-curriculum with Docker Model Runner:
```
docker model run hf.co/MatterWaveAI/grt-20m-curriculum
```

GRT-20M-Curriculum: Multi-Round Symplectic Curriculum (`MatterWaveAI/grt-20m-curriculum`)

Multi-Round Symplectic Curriculum Release: Starting from the exact same 16.7M-parameter (10.3M non-embedding backbone across 12 rotational blocks) checkpoint as MatterWaveAI/grt-20m, MatterWaveAI/grt-20m-curriculum applies multi-round curriculum optimization with HuygensSymplecticAdam. Across the 9-task SLM-Bench multiple-choice suite (13,144 held-out examples), macro-average accuracy rises from 38.63% (PairAdam NeurIPS Submission) → 45.03% (Single-Pass HuygensSymplecticAdam) → 48.17% (Multi-Round Curriculum, +9.54%), while GSM8K 4-way numerical verification reaches 98.20% (PPL = 7.65, down 7.86× from 60.12).

Model Details


Total parameters	16,730,869 (16.7M)
Non-embedding parameters	10,280,869 (10.3M)
d_model	256
n_blocks	12
n_heads	4
n_qkv_rotations	11
ffn_n_rotations	21
Vocabulary	50,000 (BPE)
Max sequence length	1,024
Precision	float32

Parameter counts are measured from the published model.safetensors. "20m" is the size-class label, not a parameter count. The embedding dominates: 6.45M of 16.7M parameters are the token embedding (50,000 tokens x 128 phase angles, plus a log-magnitude per token).

Training


Base Pretraining Data	ClimbMix, Wikipedia, RefinedWeb, FineWeb-Edu, Cosmopedia, Nemotron, FineMath, StackExchange
Base Pretraining Steps	158,000
Base Pretraining Optimizer	PairAdam (`MatterWaveAI/grt-20m-base`)
Base Peak LR	1e-3, cosine decay to 10%
Base Training Loss	2.114
Task Adaptation Optimizer	`HuygensSymplecticAdam` (`SymplecticAdam` two-stage curriculum)

Sample output (greedy, temperature 0)

The three reference completions, generated by the published weights through transformers f32, and reproduced token-for-token identically by the vLLM and llama.cpp serving paths:

Prompt	Greedy continuation (20 tokens)
`The capital of France is`	`the capital of the capital of the capital of the capital of the capital of the capital of the capital`
`Once upon a time`	`, there was a great idea to help us understand the complexities of the concept of the concept of the`
`In the beginning`	`, the first term is the same, but the second term is the same.\n\nThe problem`

This is what the model does: locally plausible, globally wrong. That is the expected behaviour of a 16.7M base model at loss 2.114, not a serving bug.

Benchmarks: NeurIPS Submission (`PairAdam`) → Single-Pass `HuygensSymplecticAdam` → Multi-Round Curriculum (`grt-20m-curriculum`)

Evaluated across the 10-task SLM-Bench small-language-model suite (training/slm_bench.py / grt-slm-bench) starting from the exact same 16.7M-parameter (10.3M non-embedding) checkpoint:

1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

Task (`SLM-Bench`)	Split (`n`)	Random	Zero-Shot	NeurIPS Baseline (`PairAdam`)	Single-Pass (`HuygensSymplectic`)	Multi-Round Curriculum	Total Gain (Curriculum vs. `PairAdam`)
BoolQ (`google/boolq`)	`val` (`3,270`)	`50.00%`	`62.17%`	`53.43%`	`62.35%`	`66.42%`	`+12.99%` (`+4.07%` vs. Single-Pass)
ARC-Easy (`allenai/ai2_arc`)	`test` (`2,376`)	`25.00%`	`26.43%`	`26.85%`	`36.70%`	`37.84%`	`+10.99%` (`+1.14%` vs. Single-Pass)
ARC-Challenge (`allenai/ai2_arc`)	`test` (`1,172`)	`25.00%`	`25.34%`	`25.43%`	`27.13%`	`30.29%`	`+4.86%` (`+3.16%` vs. Single-Pass)
OpenBookQA (`allenai/openbookqa`)	`test` (`500`)	`25.00%`	`34.20%`	`25.60%`	`37.20%`	`45.00%`	`+19.40%` (`+7.80%` vs. Single-Pass)
PIQA (`baber/piqa`)	`val` (`1,838`)	`50.00%`	`53.97%`	`53.37%`	`53.97%`	`57.13%`	`+3.76%` (`+3.16%` vs. Single-Pass)
WinoGrande (`allenai/winogrande`)	`val` (`1,267`)	`50.00%`	`51.93%`	`52.41%`	`52.41%`	`52.41%`	`+0.00%` (`+0.00%` vs. Single-Pass)
HellaSwag (`Rowan/hellaswag`)	`val` (`1,000`)	`25.00%`	`28.70%`	`31.80%`	`32.70%`	`37.50%`	`+5.70%` (`+4.80%` vs. Single-Pass)
QNLI (`nyu-mll/glue`)	`val` (`1,000`)	`50.00%`	`50.50%`	`55.70%`	`78.20%`	`78.40%`	`+22.70%` (`+0.20%` vs. Single-Pass)
CommonsenseQA (`tau/commonsense_qa`)	`val` (`1,221`)	`20.00%`	`19.16%`	`23.10%`	`24.57%`	`28.58%`	`+5.48%` (`+4.01%` vs. Single-Pass)
9-Task Multiple-Choice Macro Average	`13,144` total	`35.56%`	`39.16%`	`38.63%`	`45.03%`	`48.17%`	`+9.54%` (`+3.15%` vs. Single-Pass)

2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`, `test` split, `n = 500`)

Metric (`SLM-Bench`)	Zero-Shot	NeurIPS Baseline (`PairAdam`)	Single-Pass (`HuygensSymplectic`)	Multi-Round Curriculum	Total Improvement (Curriculum vs. Zero-Shot / `PairAdam`)
GSM8K Held-Out Perplexity (`PPL` ↓)	`60.12`	`12.52`	`9.11`	`7.65`	`7.86×` lower vs. Zero-Shot (`-38.9%` vs. `PairAdam`)
GSM8K 4-Way Numerical Verification (`Acc` ↑)	`21.00%`	`28.80%`	`93.20%`	`98.20%`	`+69.40%` vs. `PairAdam` (`+5.00%` vs. Single-Pass)

Overview of Multi-Round Curriculum Optimization

While single-pass task adaptation with HuygensSymplecticAdam already improves substantially over first-generation PairAdam, a compact 10.3M-parameter rotational backbone (12 Givens butterfly blocks on SO(D)) benefits further from multi-round curriculum training:

Norm-Preserving Rotational Geometry (SO(D)): Unlike standard dense transformers (nn.Linear) whose unconstrained weight matrices overfit small-scale benchmarks within 1–2 epochs, GRT-20M parameterizes its projections as cascades of norm-preserving 2×2 Givens rotations. Because orthogonal transformations preserve vector norms (||R(theta) x||_2 = ||x||_2), the backbone can be trained across multiple progressive curriculum stages without norm explosion or catastrophic forgetting.
Symplectic & Wavefront-Coupled Angular Updates (HuygensSymplecticAdam): SymplecticAdam and HuygensSymplecticAdam track angular velocity and cross-stage phase coherence across the Givens cascades, allowing deeper rotational blocks to align smoothly over multi-stage schedules where first-order PairAdam stalls.
Progressive Multi-Round Curriculum: Starting from the base pretrained checkpoint, the model is adapted across successive curriculum stages with cosine learning-rate scheduling, steadily refining angular phase alignment across all 12 blocks before consolidating into the published model.safetensors checkpoint.

Usage

import torch
import grt  # registers model_type "grt" with the transformers Auto classes
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-20m-curriculum", dtype=torch.float32)
tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-20m-curriculum")

input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional and load automatically when installed.

trust_remote_code=True is not supported for this repository. The model code is distributed as a pip package rather than as Hub remote code, because the implementation spans a package with absolute imports that a flat Hub snapshot cannot resolve. Install geometric-rotation-transformer first; import grt then registers model_type: "grt" so the Auto classes resolve these weights.

GGUF

grt-20m-f32.gguf is the same weights in GGUF form, loadable by the MatterWave llama.cpp fork. Verify any pull against SHA256SUMS in this repo:

sha256sum --ignore-missing -c SHA256SUMS

Architecture

GRT replaces the standard Q/K/V projections and FFN weight matrices with cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is modulated by data-dependent Clifford features:

feat  = [S0, V1, V2, B12]           # scalar energy, direction, cross-pair bivector
theta = angles + tanh(modulation)   # base angle + content-dependent nudge
r0    = x0*cos(theta) - x1*sin(theta)
r1    = x0*sin(theta) + x1*cos(theta)

Components:

RotationalEmbedding — tokens stored as phase angles on the unit hypersphere
RotationalAttention — Q/K/V via Givens cascades
RotationalFFN — feed-forward via meta-coupled rotation stages
PearlNorm — pair-radius normalization preserving per-pair phase
RadialGELU — activation gating pair magnitudes, preserving direction
PairAdam — AdamW with factored second moments: 1D params (angles, scales) use standard diagonal moments, 2D params (coupling matrices) use a row factor [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n) instead of O(mn). Optionally clamps per-step angular updates.

Because every projection is a rotation, the transform is norm-preserving by construction rather than by normalization.

Intended Use & Limitations

Intended for research into structured/rotational architectures and as a deterministic CI upstream. Not intended for production use, deployment, or any task requiring factual output.

Capabilities this model does not have — the platform metadata reflects this (supports_tools: false, supports_response_schema: false, supports_vision: false):

No instruction tuning. This is a base model trained on next-token prediction only.
No chat template. There is no chat_template in this repo; an OpenAI-compatible /v1/chat/completions endpoint serves it by concatenating the messages, which is fine for a demo and wrong for anything else.
No tool calling, no structured/schema-constrained output, no vision.
English only. The mix includes web crawl (RefinedWeb, ClimbMix), encyclopedic text, synthetic textbooks (Cosmopedia), math (FineMath) and Q&A (StackExchange), so it inherits the coverage gaps and biases of all of them.
Generated text is frequently incoherent past a clause or two. Do not expect instruction-following, factual reliability, or coherent long-form output.
No safety tuning, RLHF, or content filtering of any kind.
Context is capped at 1,024 tokens.

Citation

@article{grt2026,
  title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling},
  author={MatterWave AI},
  year={2026},
  url={https://github.com/matterwaveai/grt}
}

License

Apache 2.0

Downloads last month: 162

Safetensors

Model size

16.7M params

Tensor type

I64

F32

MatterWaveAI
/

grt-20m-curriculum

GRT-20M-Curriculum: Multi-Round Symplectic Curriculum (`MatterWaveAI/grt-20m-curriculum`)

Model Details

Training

Sample output (greedy, temperature 0)

Benchmarks: NeurIPS Submission (`PairAdam`) → Single-Pass `HuygensSymplecticAdam` → Multi-Round Curriculum (`grt-20m-curriculum`)

1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`, `test` split, `n = 500`)

Overview of Multi-Round Curriculum Optimization

Usage

GGUF

Architecture

Intended Use & Limitations

Citation

License

Datasets used to train MatterWaveAI/grt-20m-curriculum

GRT-20M-Curriculum: Multi-Round Symplectic Curriculum (MatterWaveAI/grt-20m-curriculum)

Model Details

Training

Sample output (greedy, temperature 0)

Benchmarks: NeurIPS Submission (PairAdam) → Single-Pass HuygensSymplecticAdam → Multi-Round Curriculum (grt-20m-curriculum)

1. Multiple-Choice Accuracy (9 Core SLM-Bench Tasks, 13,144 Held-Out Examples)

2. Mathematical Reasoning Held-Out Perplexity & Verification (GSM8K, test split, n = 500)

Overview of Multi-Round Curriculum Optimization

Usage

GGUF

Architecture

Intended Use & Limitations

Citation

License

Datasets used to train MatterWaveAI/grt-20m-curriculum

GRT-20M-Curriculum: Multi-Round Symplectic Curriculum (`MatterWaveAI/grt-20m-curriculum`)

Benchmarks: NeurIPS Submission (`PairAdam`) → Single-Pass `HuygensSymplecticAdam` → Multi-Round Curriculum (`grt-20m-curriculum`)

1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`, `test` split, `n = 500`)