File size: 5,776 Bytes
6f6a759 5115ebe 6f6a759 5115ebe 6f6a759 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- grt
- code
- python
- tool-calling
- function-calling
- rotational-transformer
- givens-rotations
- clifford-algebra
- matterwave
base_model: MatterWaveAI/grt-30m
---
# GRT-30M-Code (`MatterWaveAI/grt-30m-code`)
**GRT-30M-Code** is the **Python algorithmic code synthesis and structured JSON tool-calling specialist** fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`GRTForCausalLM`, `29,305,498` parameters, **zero `nn.Linear` layers**, **zero `nn.Conv1d` layers**).
It specializes the 12-layer Givens Rotational Transformer backbone for:
1. **Type-Annotated Python Algorithmic Synthesis**: Generating clean, syntactically valid (`ast.parse`) Python functions across search, graph/array dynamic programming, number theory, string parsing, and linear algebra.
2. **Structured `<tool_call>` JSON Invocation**: Emitting schema-compliant `{"name": ..., "arguments": {...}}` JSON tool invocations conditioned on developer instructions.
## Code & Tool-Calling Alignment Metrics
- **Held-Out Code & Tool-Call Validation Perplexity**: `54.93` (`val_loss = 4.0061` base) → **`1.7632`** (`val_loss = 0.5672` w/ `HuygensAdam`; `1.8253` w/ `PairAdam`, `1.8619` w/ `GyricAdam`)
- **Held-Out Python Code Token Top-1 Accuracy**: `46.80%` → **`99.07%`** (`HuygensAdam`; `98.93%` w/ `CyclideAdam`, `98.53%` w/ `GyricAdam`)
- **Held-Out Python AST Syntax Validity (`ast.parse`)**: `0.00%` → **`91.67%`** (`HuygensAdam`; `83.33%` w/ `CyclideAdam`)
- **Held-Out Structured Tool-Call Token Top-1 Accuracy**: `36.44%` → **`90.69%`** (`HuygensAdam` & `PairAdam`; `89.50%` w/ `CyclideAdam`)
- **Held-Out JSON Tool-Call Schema Match Rate**: `0.00%` → **`83.33%`** (`HuygensAdam` & `PairAdam`; `75.00%` w/ `CyclideAdam`, `GyricAdam`, `HuygensSymplecticAdam`)
- **Code Signature/Docstring Conditioning Margin (`ΔL`)**: `0.8998` → **`1.6597` nats/tok** (`PairAdam`) / **`1.5766` nats/tok** (`SymplecticAdam`)
---
## 8-Way Classical Geometric Optimizer Ablation
We benchmarked all 8 classical geometric, Riemannian, conic, symplectic, gyroscopic, and synchronization optimizers from `transformers.models.grt.optimization_grt` on `MatterWaveAI/grt-30m-code` (`200` SFT steps on `blocks[11:12]` + `final_norm` + `<|im_end|>` polar phase calibration, `811,907` trainable parameters, `batch_size = 16`, `lr = 6.5e-3`, `max_angular_step = 0.12`, `seed = 42`):
| Rank | Optimizer (`optimizer_type`) | Final Train Loss | Val Loss | Val PPL | Code Token Acc (%) | Tool-Call Token Acc (%) | JSON Schema Match (%) | Python AST Valid (%) | Conditioning Margin (`ΔL`) | Wall Time (s) |
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **1** | **`HuygensAdam` (`huygens_adam`)** | **`1.0882`** | **`0.5672`** | **`1.7632`** | **`99.07%`** | **`90.69%`** | **`83.33%`** | **`91.67%`** | `1.5152` | `138.63s` |
| **2** | **`PairAdam` (`pair_adam`)** | `1.0994` | `0.6018` | `1.8253` | `97.60%` | **`90.69%`** | **`83.33%`** | `50.00%` | **`1.6597`** | `278.80s` |
| **3** | **`GyricAdam` (`gyric_adam`)** | `1.1168` | `0.6216` | `1.8619` | `98.53%` | `87.92%` | `75.00%` | `50.00%` | `1.5745` | `138.83s` |
| **4** | **`CyclideAdam` (`cyclide_adam`)** | `1.1564` | `0.6400` | `1.8965` | `98.93%` | `89.50%` | `75.00%` | `83.33%` | `1.5727` | `141.92s` |
| **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `1.1819` | `0.6537` | `1.9227` | `98.27%` | `89.31%` | `75.00%` | `66.67%` | `1.4057` | `122.82s` |
| **6** | **`HessianPairAdam` (`hessian_pair_adam`)** | `1.2171` | `0.7292` | `2.0734` | `97.87%` | `87.33%` | `66.67%` | `58.33%` | `1.4236` | `117.82s` |
| **7** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `1.2463` | `0.7891` | `2.2015` | `96.93%` | `86.34%` | `66.67%` | `50.00%` | `1.4294` | **`108.18s`** |
| **8** | **`SymplecticAdam` (`symplectic_adam`)** | `1.2764` | `0.8445` | `2.3268` | `97.60%` | `81.39%` | `50.00%` | `50.00%` | `1.5766` | `138.10s` |
### Key Geometric Findings on Python Code & Structured Tool Calling
- **Huygens Circular Synchronization Splay (`HuygensAdam`) Sweeps Code & Tool-Call Generation**: By preventing butterfly rotation stages from collapsing onto repetitive indentation/delimiter tokens, `HuygensAdam` achieves **#1 validation loss (`0.5672` / `1.7632` PPL)**, **#1 Python code token accuracy (`99.07%`)**, **#1 Python AST syntax validity (`91.67%`)**, and **#1 tool-call token accuracy (`90.69%`) & JSON schema match rate (`83.33%`)**.
- **Dupin Cyclide Conic Metric (`CyclideAdam`) Excels at Syntactic Delimiter Balance**: `CyclideAdam` achieves **`98.93%` Python code token accuracy** and **`83.33%` Python AST syntax validity** (#2 overall on AST validity).
---
## Quickstart (`transformers`)
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "MatterWaveAI/grt-30m-code"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32)
prompt = (
"<|im_start|>system\n"
"You are GRT-30M-Code, a Python algorithmic coding and structured JSON tool-calling specialist "
"built by MatterWave AI on the 100% rotational Givens Rotational Transformer architecture.<|im_end|>\n"
"<|im_start|>user\n"
"Write a Python function `gcd(a: int, b: int) -> int` using Euclid's iterative algorithm.<|im_end|>\n"
"<|im_start|>assistant\n"
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
out_ids = model.generate(**inputs, max_new_tokens=48, do_sample=False)
print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
---
## License
Apache 2.0
|