davidhouseholter commited on
Commit
aae99bf
·
verified ·
1 Parent(s): ec803d2

Release MatterWaveAI/grt-30m-lora with GGUF & SHA256SUMS (T-018 / T-019)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ grt-30m-lora-f16.gguf filter=lfs diff=lfs merge=lfs -text
37
+ grt-30m-lora-f32.gguf filter=lfs diff=lfs merge=lfs -text
38
+ GRT_Whitepaper.pdf filter=lfs diff=lfs merge=lfs -text
39
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
GRT_Whitepaper.pdf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:79219257df919cc598a75f7ae0968988ed0d33ffc1777469f9a9da39a558819d
3
+ size 1034984
PROVENANCE.md ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ # Provenance: `MatterWaveAI/grt-30m-lora`
2
+
3
+ - **Base Model**: `MatterWaveAI/grt-30m` (`release/30m/model.safetensors`)
4
+ - **Adapter Class**: `GRTRotationalLoRA` (`r=8`, `lora_alpha=16.0`, `tune_scales=True`)
5
+ - **Optimizer**: `HessianPairAdam` (`Sophia-Torus`, `lr=1.5e-2`, `80` steps)
6
+ - **Build Script**: `release/build_30m_lora.py`
7
+ - **Before Adaptation**: loss `6.172`, perplexity `479.1388`
8
+ - **After Adaptation**: loss `4.7464`, perplexity `115.1645`
9
+ - **Merge Equivalence**: `max_abs_diff = 0.0e+00` (`0` `nn.Linear`, `0` `nn.Conv1d`)
README.md ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - grt
7
+ - lora
8
+ - peft
9
+ - toroidal-lora
10
+ - rotational-transformer
11
+ - givens-rotations
12
+ - clifford-algebra
13
+ - sophia-torus
14
+ - matterwave
15
+ base_model: MatterWaveAI/grt-30m
16
+ ---
17
+
18
+ # GRT-30M-LoRA (`MatterWaveAI/grt-30m-lora`)
19
+
20
+ **GRT-30M-LoRA** is the canonical **Toroidal Low-Rank Adaptation (`GRTRotationalLoRA`)** release for [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` base parameters, **`0` `nn.Linear` layers**, **`0` `nn.Conv1d` layers**).
21
+
22
+ Because every projection in a Geometric Rotation Transformer is parameterized by phase angles on the flat torus $\mathbb{T}^P = (S^1)^P$ and radial scales rather than dense Euclidean weight matrices, standard linear LoRA ($W + BA$) cannot be applied without violating orthogonal/unitary manifold invariants. **`GRTRotationalLoRA`** instead attaches low-rank phase-shift matrices `lora_A` and `lora_B` directly to the butterfly rotation angle tensors:
23
+
24
+ $$\theta = \left(\theta_{\text{base}} + \frac{\alpha}{r}\, A B\right) \bmod 2\pi$$
25
+
26
+ At inference time, `merge_grt_lora(model)` folds $\Delta\theta = \frac{\alpha}{r} A B$ directly into $\theta_{\text{base}}$ with **bit-exact equivalence** (`max_abs_diff = 0.0`) and **zero runtime latency or parameter overhead**.
27
+
28
+ ---
29
+
30
+ ## Key Highlights & Metrics
31
+
32
+ | Metric | Value |
33
+ | :--- | :--- |
34
+ | **Adapter Class** | `GRTRotationalLoRA` (`transformers.models.grt.lora_grt`) |
35
+ | **Base Checkpoint** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`29,305,498` params) |
36
+ | **LoRA Rank ($r$) / Alpha ($\alpha$)** | `r = 8`, `lora_alpha = 16.0` (`scaling = 2.0`) |
37
+ | **Trainable LoRA + Scale Parameters** | **`140,741`** (`0.48%` of base model) |
38
+ | **Adapter Checkpoint Size** | **`573.3 KB`** (`grt_lora_adapter.safetensors`) |
39
+ | **`nn.Linear` / `nn.Conv1d` Layers** | **`0` / `0` (100% Rotational Purity)** |
40
+ | **Optimizer** | `HessianPairAdam` (`Sophia-Torus` analytic Gauss-Newton curvature) |
41
+ | **Domain Assistant Loss (Before $\rightarrow$ After)** | `6.1720` $\rightarrow$ **`4.7464`** |
42
+ | **Domain Perplexity (Before $\rightarrow$ After)** | `479.14` $\rightarrow$ **`115.16`** |
43
+ | **Token Accuracy (Before $\rightarrow$ After)** | `19.5%` $\rightarrow$ **`28.8%`** |
44
+ | **Pre-Merge vs. Post-Merge `max_abs_diff`** | **`0.0e+00`** (bit-exact) |
45
+
46
+ ---
47
+
48
+ ## Quickstart (`transformers`)
49
+
50
+ ### 1. Load & Merge `MatterWaveAI/grt-30m-lora` into `MatterWaveAI/grt-30m`
51
+
52
+ ```python
53
+ import torch
54
+ from huggingface_hub import snapshot_download
55
+ from transformers import AutoTokenizer
56
+ from transformers.models.grt import GRTForCausalLM, load_grt_lora_pretrained
57
+
58
+ base_id = "MatterWaveAI/grt-30m"
59
+ adapter_dir = snapshot_download("MatterWaveAI/grt-30m-lora")
60
+
61
+ tokenizer = AutoTokenizer.from_pretrained(base_id)
62
+ model = GRTForCausalLM.from_pretrained(base_id, torch_dtype=torch.float32)
63
+
64
+ # Attach and fold the toroidal LoRA phase shifts into base_angles (0 inference overhead)
65
+ load_grt_lora_pretrained(model, adapter_dir, merge=True)
66
+
67
+ prompt = "<|im_start|>user\nExplain how GRTRotationalLoRA adapts a Geometric Rotation Transformer without linear layers.<|im_end|>\n<|im_start|>assistant\n"
68
+ inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
69
+ out_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)
70
+ print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
71
+ ```
72
+
73
+ ### 2. Train Your Own Toroidal LoRA Adapter
74
+
75
+ ```python
76
+ from transformers.models.grt import (
77
+ GRTForCausalLM,
78
+ attach_grt_lora,
79
+ build_grt_optimizer,
80
+ merge_grt_lora,
81
+ save_grt_lora_pretrained,
82
+ )
83
+
84
+ model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-30m")
85
+ attach_grt_lora(model, r=8, lora_alpha=16.0, tune_scales=True)
86
+
87
+ optimizer = build_grt_optimizer(model, lr=1e-2, weight_decay=0.0, hessian_mode=True)
88
+ # ... training loop ...
89
+ save_grt_lora_pretrained(model, "./my-grt-lora")
90
+ merge_grt_lora(model) # zero-overhead merged inference
91
+ ```
92
+
93
+ ---
94
+
95
+ ## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
96
+
97
+ 1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
98
+ 2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
99
+ 3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
100
+ 4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
101
+ 5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
102
+ 6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
103
+ 7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
104
+ 8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
105
+ 9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
106
+ 10. **Toroidal Rotational LoRA (`GRTRotationalLoRA`)**: [`MatterWaveAI/grt-30m-lora`](https://huggingface.co/MatterWaveAI/grt-30m-lora)
107
+ 11. **Rotational Speech-to-Text ASR (`GRTASR`)**: [`MatterWaveAI/grt-30m-asr`](https://huggingface.co/MatterWaveAI/grt-30m-asr)
108
+
109
+ - **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
110
+
111
+ ---
112
+
113
+ ## License
114
+
115
+ Apache 2.0
116
+
117
+ ---
118
+
119
+ ## Citation
120
+
121
+ ```bibtex
122
+ @techreport{householter2026grt,
123
+ title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
124
+ author = {Householter, David},
125
+ institution = {MatterWave AI},
126
+ year = {2026},
127
+ url = {https://matterwave.ai}
128
+ }
129
+ ```
SHA256SUMS ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ 90d026638fdf7b4cc1269e64c391b9000cbe2fa2c356a0684378d6fac7119460 eval_results.json
2
+ 000bafd40f80aeddca312d8afea332b4a8f2bb73e3421339474974e6a027cfb6 grt-30m-lora-f16.gguf
3
+ 72ae91fdcb81d98e9deff84930baf9f937cbd7c5c8ae577c4544d05f11677711 grt-30m-lora-f32.gguf
4
+ 6f82a2c5487a9b7b002e0809f8b9f6ec045b69051671ca6ae72d094a633c5a00 grt_lora_adapter.safetensors
5
+ e36913871e23ca265f3d780677eda89d6a216d7b48356964875bc7b071d54ad2 grt_lora_config.json
6
+ 79219257df919cc598a75f7ae0968988ed0d33ffc1777469f9a9da39a558819d GRT_Whitepaper.pdf
7
+ 8b78063b3dd750d7b6b33513b7e1f49a1e88019e74097f1282d1a48e11894653 README.md
8
+ be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 tokenizer.json
9
+ b0b05688d53eec97348d23f9a203772fa28f12bdc8a0f6f5401f41568ec6ca31 tokenizer_config.json
eval_results.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_id": "MatterWaveAI/grt-30m-lora",
3
+ "adapter_class": "GRTRotationalLoRA",
4
+ "base_model": "MatterWaveAI/grt-30m",
5
+ "optimizer": "HessianPairAdam (Sophia-Torus)",
6
+ "base_parameters": 29305498,
7
+ "trainable_parameters": 140741,
8
+ "total_parameters_with_lora": 29436730,
9
+ "trainable_pct": 0.4803,
10
+ "lora_rank": 8,
11
+ "lora_alpha": 16.0,
12
+ "lora_scaling": 2.0,
13
+ "nn_linear_layers": 0,
14
+ "nn_conv1d_layers": 0,
15
+ "step0_identity_max_abs_diff": 0.0,
16
+ "merge_max_abs_diff": 0.0,
17
+ "reload_max_abs_diff": 0.0,
18
+ "adapter_file_bytes": 587108,
19
+ "training_steps": 80,
20
+ "before_adaptation": {
21
+ "assistant_loss": 6.172,
22
+ "perplexity": 479.1388,
23
+ "token_accuracy": 0.1954
24
+ },
25
+ "after_adaptation": {
26
+ "assistant_loss": 4.7464,
27
+ "perplexity": 115.1645,
28
+ "token_accuracy": 0.2883
29
+ },
30
+ "sample_prompt": "Explain how GRTRotationalLoRA adapts a Geometric Rotation Transformer without linear layers.",
31
+ "sample_response": "In each 2. equal 2^12^1^1^1^1^1^1^1^1^1^1^1^1^1^1^1^1^1^1^"
32
+ }
grt-30m-lora-f16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:000bafd40f80aeddca312d8afea332b4a8f2bb73e3421339474974e6a027cfb6
3
+ size 64554240
grt-30m-lora-f32.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72ae91fdcb81d98e9deff84930baf9f937cbd7c5c8ae577c4544d05f11677711
3
+ size 356546592
grt_lora_adapter.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f82a2c5487a9b7b002e0809f8b9f6ec045b69051671ca6ae72d094a633c5a00
3
+ size 587108
grt_lora_config.json ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "peft_type": "GRT_ROTATIONAL_LORA",
3
+ "adapter_type": "GRTRotationalLoRA",
4
+ "auto_mapping": {
5
+ "base_model_class": "GRTForCausalLM"
6
+ },
7
+ "base_model_name_or_path": "MatterWaveAI/grt-30m",
8
+ "r": 8,
9
+ "lora_alpha": 16.0,
10
+ "scaling": 2.0,
11
+ "lora_dropout": 0.0,
12
+ "target_modules": [
13
+ "q_angles",
14
+ "k_angles",
15
+ "v_angles",
16
+ "o_angles",
17
+ "expand_angles",
18
+ "contract_angles",
19
+ "coupling_angles"
20
+ ],
21
+ "tune_scales": true,
22
+ "manifold": "T^P = (S^1)^P",
23
+ "formula": "theta = (base_angles + (lora_alpha / r) * (lora_A @ lora_B)) mod 2*pi",
24
+ "zero_linear_layers": true,
25
+ "num_adapter_tensors": 242,
26
+ "num_adapter_parameters": 140741,
27
+ "model_id": "MatterWaveAI/grt-30m-lora",
28
+ "optimizer": "HessianPairAdam (Sophia-Torus)",
29
+ "training_steps": 80,
30
+ "before_adaptation": {
31
+ "assistant_loss": 6.172,
32
+ "perplexity": 479.1388,
33
+ "token_accuracy": 0.1954
34
+ },
35
+ "after_adaptation": {
36
+ "assistant_loss": 4.7464,
37
+ "perplexity": 115.1645,
38
+ "token_accuracy": 0.2883
39
+ }
40
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
3
+ size 11422650
tokenizer_config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "is_local": true,
24
+ "local_files_only": false,
25
+ "model_max_length": 131072,
26
+ "pad_token": "<|endoftext|>",
27
+ "split_special_tokens": false,
28
+ "tokenizer_class": "Qwen2Tokenizer",
29
+ "unk_token": null
30
+ }