| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - grt |
| - code |
| - python |
| - tool-calling |
| - function-calling |
| - rotational-transformer |
| - givens-rotations |
| - clifford-algebra |
| - matterwave |
| base_model: MatterWaveAI/grt-30m |
| --- |
| |
| # GRT-30M-Code (`MatterWaveAI/grt-30m-code`) |
|
|
| **GRT-30M-Code** is the **Python algorithmic code synthesis and structured JSON tool-calling specialist** fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) (`GRTForCausalLM`, `29,305,498` parameters, **zero `nn.Linear` layers**, **zero `nn.Conv1d` layers**). |
|
|
| It specializes the 12-layer Givens Rotational Transformer backbone for: |
| 1. **Type-Annotated Python Algorithmic Synthesis**: Generating clean, syntactically valid (`ast.parse`) Python functions across search, graph/array dynamic programming, number theory, string parsing, and linear algebra. |
| 2. **Structured `<tool_call>` JSON Invocation**: Emitting schema-compliant `{"name": ..., "arguments": {...}}` JSON tool invocations conditioned on developer instructions. |
|
|
| ## Code & Tool-Calling Alignment Metrics |
|
|
| - **Held-Out Code & Tool-Call Validation Perplexity**: `54.93` (`val_loss = 4.0061` base) → **`1.7632`** (`val_loss = 0.5672` w/ `HuygensAdam`; `1.8253` w/ `PairAdam`, `1.8619` w/ `GyricAdam`) |
| - **Held-Out Python Code Token Top-1 Accuracy**: `46.80%` → **`99.07%`** (`HuygensAdam`; `98.93%` w/ `CyclideAdam`, `98.53%` w/ `GyricAdam`) |
| - **Held-Out Python AST Syntax Validity (`ast.parse`)**: `0.00%` → **`91.67%`** (`HuygensAdam`; `83.33%` w/ `CyclideAdam`) |
| - **Held-Out Structured Tool-Call Token Top-1 Accuracy**: `36.44%` → **`90.69%`** (`HuygensAdam` & `PairAdam`; `89.50%` w/ `CyclideAdam`) |
| - **Held-Out JSON Tool-Call Schema Match Rate**: `0.00%` → **`83.33%`** (`HuygensAdam` & `PairAdam`; `75.00%` w/ `CyclideAdam`, `GyricAdam`, `HuygensSymplecticAdam`) |
| - **Code Signature/Docstring Conditioning Margin (`ΔL`)**: `0.8998` → **`1.6597` nats/tok** (`PairAdam`) / **`1.5766` nats/tok** (`SymplecticAdam`) |
|
|
| --- |
|
|
| ## 8-Way Classical Geometric Optimizer Ablation |
|
|
| We benchmarked all 8 classical geometric, Riemannian, conic, symplectic, gyroscopic, and synchronization optimizers from `transformers.models.grt.optimization_grt` on `MatterWaveAI/grt-30m-code` (`200` SFT steps on `blocks[11:12]` + `final_norm` + `<|im_end|>` polar phase calibration, `811,907` trainable parameters, `batch_size = 16`, `lr = 6.5e-3`, `max_angular_step = 0.12`, `seed = 42`): |
|
|
| | Rank | Optimizer (`optimizer_type`) | Final Train Loss | Val Loss | Val PPL | Code Token Acc (%) | Tool-Call Token Acc (%) | JSON Schema Match (%) | Python AST Valid (%) | Conditioning Margin (`ΔL`) | Wall Time (s) | |
| | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | |
| | **1** | **`HuygensAdam` (`huygens_adam`)** | **`1.0882`** | **`0.5672`** | **`1.7632`** | **`99.07%`** | **`90.69%`** | **`83.33%`** | **`91.67%`** | `1.5152` | `138.63s` | |
| | **2** | **`PairAdam` (`pair_adam`)** | `1.0994` | `0.6018` | `1.8253` | `97.60%` | **`90.69%`** | **`83.33%`** | `50.00%` | **`1.6597`** | `278.80s` | |
| | **3** | **`GyricAdam` (`gyric_adam`)** | `1.1168` | `0.6216` | `1.8619` | `98.53%` | `87.92%` | `75.00%` | `50.00%` | `1.5745` | `138.83s` | |
| | **4** | **`CyclideAdam` (`cyclide_adam`)** | `1.1564` | `0.6400` | `1.8965` | `98.93%` | `89.50%` | `75.00%` | `83.33%` | `1.5727` | `141.92s` | |
| | **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `1.1819` | `0.6537` | `1.9227` | `98.27%` | `89.31%` | `75.00%` | `66.67%` | `1.4057` | `122.82s` | |
| | **6** | **`HessianPairAdam` (`hessian_pair_adam`)** | `1.2171` | `0.7292` | `2.0734` | `97.87%` | `87.33%` | `66.67%` | `58.33%` | `1.4236` | `117.82s` | |
| | **7** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `1.2463` | `0.7891` | `2.2015` | `96.93%` | `86.34%` | `66.67%` | `50.00%` | `1.4294` | **`108.18s`** | |
| | **8** | **`SymplecticAdam` (`symplectic_adam`)** | `1.2764` | `0.8445` | `2.3268` | `97.60%` | `81.39%` | `50.00%` | `50.00%` | `1.5766` | `138.10s` | |
| |
| ### Key Geometric Findings on Python Code & Structured Tool Calling |
| - **Huygens Circular Synchronization Splay (`HuygensAdam`) Sweeps Code & Tool-Call Generation**: By preventing butterfly rotation stages from collapsing onto repetitive indentation/delimiter tokens, `HuygensAdam` achieves **#1 validation loss (`0.5672` / `1.7632` PPL)**, **#1 Python code token accuracy (`99.07%`)**, **#1 Python AST syntax validity (`91.67%`)**, and **#1 tool-call token accuracy (`90.69%`) & JSON schema match rate (`83.33%`)**. |
| - **Dupin Cyclide Conic Metric (`CyclideAdam`) Excels at Syntactic Delimiter Balance**: `CyclideAdam` achieves **`98.93%` Python code token accuracy** and **`83.33%` Python AST syntax validity** (#2 overall on AST validity). |
| |
| --- |
| |
| ## Quickstart (`transformers`) |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "MatterWaveAI/grt-30m-code" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32) |
| |
| prompt = ( |
| "<|im_start|>system\n" |
| "You are GRT-30M-Code, a Python algorithmic coding and structured JSON tool-calling specialist " |
| "built by MatterWave AI on the 100% rotational Givens Rotational Transformer architecture.<|im_end|>\n" |
| "<|im_start|>user\n" |
| "Write a Python function `gcd(a: int, b: int) -> int` using Euclid's iterative algorithm.<|im_end|>\n" |
| "<|im_start|>assistant\n" |
| ) |
| inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False) |
| out_ids = model.generate(**inputs, max_new_tokens=48, do_sample=False) |
| print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
| |
| --- |
| |
| ## License |
| |
| Apache 2.0 |
| |
