Instructions to use MatterWaveAI/grt-20m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MatterWaveAI/grt-20m with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MatterWaveAI/grt-20m")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-20m", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use MatterWaveAI/grt-20m with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf MatterWaveAI/grt-20m:F32
# Run inference directly in the terminal:
llama cli -hf MatterWaveAI/grt-20m:F32

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf MatterWaveAI/grt-20m:F32
# Run inference directly in the terminal:
llama cli -hf MatterWaveAI/grt-20m:F32

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf MatterWaveAI/grt-20m:F32
# Run inference directly in the terminal:
./build/bin/llama cli -hf MatterWaveAI/grt-20m:F32

Use Docker

docker model run hf.co/MatterWaveAI/grt-20m:F32

Llama new
LM Studio
Jan

vLLM

How to use MatterWaveAI/grt-20m with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MatterWaveAI/grt-20m"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MatterWaveAI/grt-20m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MatterWaveAI/grt-20m:F32

SGLang

How to use MatterWaveAI/grt-20m with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MatterWaveAI/grt-20m" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MatterWaveAI/grt-20m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MatterWaveAI/grt-20m" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MatterWaveAI/grt-20m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use MatterWaveAI/grt-20m with Ollama:
```
ollama run hf.co/MatterWaveAI/grt-20m:F32
```
Unsloth Desktop
Docker Model Runner
How to use MatterWaveAI/grt-20m with Docker Model Runner:
```
docker model run hf.co/MatterWaveAI/grt-20m:F32
```

Lemonade

How to use MatterWaveAI/grt-20m with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull MatterWaveAI/grt-20m:F32

Run and chat with the model

lemonade run user.grt-20m-F32

List all available models

lemonade list

Atomic Chat

GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization

Note: GRT-20M (16.7M total parameters, 10.3M non-embedding backbone parameters across 12 rotational blocks) was pretrained across an 8-corpus English mix using first-generation PairAdam for our NeurIPS submission (archived pre-fine-tune at MatterWaveAI/grt-20m-base). Keeping the exact same 16.7M NeurIPS submission checkpoint and swapping in our new classical geometric optimizers (SymplecticAdam and HuygensSymplecticAdam) lifts 9-task SLM-Bench multiple-choice accuracy from 38.63% (PairAdam) → 45.03% (HuygensSymplecticAdam, +6.40%) and cuts GSM8K held-out math perplexity from 60.12 → 9.11 (6.60× reduction vs. Zero-Shot, -27.2% vs. PairAdam). For the multi-round curriculum checkpoint (48.17% 9-task MC average, 98.20% GSM8K verification, 7.65 PPL), see MatterWaveAI/grt-20m-curriculum.

A Geometric Rotation Transformer pretrained on a mix of eight English corpora, including FineWeb-Edu, Cosmopedia, and FineMath. GRT replaces the dense linear projections of a standard transformer with cascades of learned Givens rotations on SO(D), so the residual path contains no nn.Linear at all.

This is a research preview, not a product model. It exists to show that the architecture trains stably and evaluates end-to-end through the same OpenAI-HTTP serving seam as every other model, and to give CPU-only CI a cheap, always- available upstream. At this scale the output is grammatical and locally coherent but not factual — see the real samples below before expecting anything else. It is not instruction-tuned and has no chat template.

Model Details


Total parameters	16,730,869 (16.7M)
Non-embedding parameters	10,280,869 (10.3M)
d_model	256
n_blocks	12
n_heads	4
n_qkv_rotations	11
ffn_n_rotations	21
Vocabulary	50,000 (BPE)
Max sequence length	1,024
Precision	float32

Parameter counts are measured from the published model.safetensors. "20m" is the size-class label, not a parameter count. The embedding dominates: 6.45M of 16.7M parameters are the token embedding (50,000 tokens x 128 phase angles, plus a log-magnitude per token).

Training


Base Pretraining Data	ClimbMix, Wikipedia, RefinedWeb, FineWeb-Edu, Cosmopedia, Nemotron, FineMath, StackExchange
Base Pretraining Steps	158,000
Base Pretraining Optimizer	PairAdam (`MatterWaveAI/grt-20m-base`)
Base Peak LR	1e-3, cosine decay to 10%
Base Training Loss	2.114
Task Adaptation Optimizer	`HuygensSymplecticAdam` (`SymplecticAdam` two-stage curriculum)

Sample output (greedy, temperature 0)

The three reference completions, generated by the published weights through transformers f32, and reproduced token-for-token identically by the vLLM and llama.cpp serving paths:

Prompt	Greedy continuation (20 tokens)
`The capital of France is`	`the capital of the capital of the capital of the capital of the capital of the capital of the capital`
`Once upon a time`	`, there was a great idea to help us understand the complexities of the concept of the concept of the`
`In the beginning`	`, the first term is the same, but the second term is the same.\n\nThe problem`

This is what the model does: locally plausible, globally wrong. That is the expected behaviour of a 16.7M base model at loss 2.114, not a serving bug.

Benchmarks: NeurIPS Submission Baseline (`PairAdam`) vs. New Optimizer (`HuygensSymplecticAdam`)

Evaluated on the SLM-Bench small-language-model suite (training/slm_bench.py / grt-slm-bench) starting from the exact same 16.7M-parameter (10.3M non-embedding) NeurIPS submission checkpoint:

1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

Task (`SLM-Bench`)	Split (`n`)	Random	`GRT-20M` Zero-Shot	NeurIPS Baseline (`PairAdam`)	`SymplecticAdam`	New Optimizer (`HuygensSymplecticAdam`)	Gain vs. `PairAdam`
QNLI (`nyu-mll/glue`)	`val` (`1,000`)	`50.00%`	`50.50%`	`55.70%`	`74.00%`	`78.20%`	`+22.50%` (`+28.20%` vs. random)
BoolQ (`google/boolq`)	`val` (`3,270`)	`50.00%`	`62.17%`	`53.43%`	`62.17%`	`62.35%`	`+8.92%` (`+12.35%` vs. random)
PIQA (`baber/piqa`)	`val` (`1,838`)	`50.00%`	`53.97%`	`53.37%`	`53.97%`	`53.97%`	`+0.60%` (`+3.97%` vs. random)
WinoGrande (`allenai/winogrande`)	`val` (`1,267`)	`50.00%`	`51.93%`	`52.41%`	`52.41%`	`52.41%`	`+0.48%` vs. zero-shot
OpenBookQA (`allenai/openbookqa`)	`test` (`500`)	`25.00%`	`34.20%`	`25.60%`	`36.00%`	`37.20%`	`+11.60%` (`+12.20%` vs. random)
ARC-Easy (`allenai/ai2_arc`)	`test` (`2,376`)	`25.00%`	`26.43%`	`26.85%`	`34.30%`	`36.70%`	`+9.85%` (`+11.70%` vs. random)
HellaSwag (`Rowan/hellaswag`)	`val` (`1,000`)	`25.00%`	`28.70%`	`31.80%`	`32.70%`	`32.70%`	`+0.90%` (`+7.70%` vs. random)
ARC-Challenge (`allenai/ai2_arc`)	`test` (`1,172`)	`25.00%`	`25.34%`	`25.43%`	`26.79%`	`27.13%`	`+1.70%` (`+2.13%` vs. random)
CommonsenseQA (`tau/commonsense_qa`)	`val` (`1,221`)	`20.00%`	`19.16%`	`23.10%`	`23.91%`	`24.57%`	`+1.47%` (`+5.41%` vs. zero-shot)
9-Task Multiple-Choice Macro Average	`13,144` total	`35.56%`	`39.16%`	`38.63%`	`44.03%`	`45.03%`	`+6.40%` (`+9.47%` vs. random)

2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`)

Task (`SLM-Bench`)	Split (`n`)	`GRT-20M` Zero-Shot	NeurIPS Baseline (`PairAdam`)	`SymplecticAdam`	New Optimizer (`HuygensSymplecticAdam`)	Improvement
GSM8K Held-Out Perplexity (`PPL` ↓)	`test` (`500`)	`60.12`	`12.52`	`10.70`	`9.11`	`6.60×` lower vs. Zero-Shot (`-27.2%` vs. `PairAdam`)
GSM8K 4-Way Numerical Verification (`Acc` ↑)	`test` (`500`)	`21.00%`	`28.80%`	`91.40%`	`93.20%`	`+64.40%` vs. `PairAdam`

Reproduce with grt-slm-bench:

python -m training.slm_bench --checkpoint ./ --optimizer huygens_symplectic_adam \
    --tasks boolq,arc_easy,arc_challenge,openbookqa,piqa,winogrande,hellaswag,qnli,commonsenseqa,gsm8k \
    --device cuda

Zero-shot, via lm-evaluation-harness:

lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-20m \
    --tasks boolq,arc_easy --batch_size 64

Usage

import torch
import grt  # registers model_type "grt" with the transformers Auto classes
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-20m", dtype=torch.float32)
tokenizer = AutoTokenizer.from_pretrained("MatterWaveAI/grt-20m")

input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional and load automatically when installed.

trust_remote_code=True is not supported for this repository. The model code is distributed as a pip package rather than as Hub remote code, because the implementation spans a package with absolute imports that a flat Hub snapshot cannot resolve. Install geometric-rotation-transformer first; import grt then registers model_type: "grt" so the Auto classes resolve these weights.

GGUF

grt-20m-f32.gguf is the same weights in GGUF form, loadable by the MatterWave llama.cpp fork. Verify any pull against SHA256SUMS in this repo:

sha256sum --ignore-missing -c SHA256SUMS

Architecture

GRT replaces the standard Q/K/V projections and FFN weight matrices with cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is modulated by data-dependent Clifford features:

feat  = [S0, V1, V2, B12]           # scalar energy, direction, cross-pair bivector
theta = angles + tanh(modulation)   # base angle + content-dependent nudge
r0    = x0*cos(theta) - x1*sin(theta)
r1    = x0*sin(theta) + x1*cos(theta)

Components:

RotationalEmbedding — tokens stored as phase angles on the unit hypersphere
RotationalAttention — Q/K/V via Givens cascades
RotationalFFN — feed-forward via meta-coupled rotation stages
PearlNorm — pair-radius normalization preserving per-pair phase
RadialGELU — activation gating pair magnitudes, preserving direction
PairAdam — AdamW with factored second moments: 1D params (angles, scales) use standard diagonal moments, 2D params (coupling matrices) use a row factor [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n) instead of O(mn). Optionally clamps per-step angular updates.

Because every projection is a rotation, the transform is norm-preserving by construction rather than by normalization.

Intended Use & Limitations

Intended for research into structured/rotational architectures and as a deterministic CI upstream. Not intended for production use, deployment, or any task requiring factual output.

Capabilities this model does not have — the platform metadata reflects this (supports_tools: false, supports_response_schema: false, supports_vision: false):

No instruction tuning. This is a base model trained on next-token prediction only.
No chat template. There is no chat_template in this repo; an OpenAI-compatible /v1/chat/completions endpoint serves it by concatenating the messages, which is fine for a demo and wrong for anything else.
No tool calling, no structured/schema-constrained output, no vision.
English only. The mix includes web crawl (RefinedWeb, ClimbMix), encyclopedic text, synthetic textbooks (Cosmopedia), math (FineMath) and Q&A (StackExchange), so it inherits the coverage gaps and biases of all of them.
Generated text is frequently incoherent past a clause or two. Do not expect instruction-following, factual reliability, or coherent long-form output.
No safety tuning, RLHF, or content filtering of any kind.
Context is capped at 1,024 tokens.

Citation

@article{grt2026,
  title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling},
  author={MatterWave AI},
  year={2026},
  url={https://github.com/matterwaveai/grt}
}

License

Apache 2.0

Downloads last month: 442

Safetensors

Model size

16.7M params

Tensor type

I64

F32

Model tree for MatterWaveAI/grt-20m

Finetunes

3 models

MatterWaveAI
/

grt-20m

GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization

Model Details

Training

Sample output (greedy, temperature 0)

Benchmarks: NeurIPS Submission Baseline (`PairAdam`) vs. New Optimizer (`HuygensSymplecticAdam`)

1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`)

Usage

GGUF

Architecture

Intended Use & Limitations

Citation

License

Model tree for MatterWaveAI/grt-20m

Datasets used to train MatterWaveAI/grt-20m

GRT-20M: From NeurIPS Submission (PairAdam) to Symplectic/Huygens Optimization

Model Details

Training

Sample output (greedy, temperature 0)

Benchmarks: NeurIPS Submission Baseline (PairAdam) vs. New Optimizer (HuygensSymplecticAdam)

1. Multiple-Choice Accuracy (9 Core SLM-Bench Tasks, 13,144 Held-Out Examples)

2. Mathematical Reasoning Held-Out Perplexity & Verification (GSM8K)

Usage

GGUF

Architecture

Intended Use & Limitations

Citation

License

Model tree for MatterWaveAI/grt-20m

Datasets used to train MatterWaveAI/grt-20m

GRT-20M: From NeurIPS Submission (`PairAdam`) to Symplectic/Huygens Optimization

Benchmarks: NeurIPS Submission Baseline (`PairAdam`) vs. New Optimizer (`HuygensSymplecticAdam`)

1. Multiple-Choice Accuracy (`9` Core `SLM-Bench` Tasks, `13,144` Held-Out Examples)

2. Mathematical Reasoning Held-Out Perplexity & Verification (`GSM8K`)